How to make AI more responsible

Note: Transcripts are automatically generated from episode audio, and are not fully corrected for spelling, grammar, and formatting.

NAVRINA SINGH: I’ve always been a renegade. I always was challenging the status quo, always at the front end of innovation. And when I saw this big hairy problem of responsible AI, I think it was this moment of reckoning. If there was a legacy to be left and if there was a moment, that moment is now.

RANA EL KALIOUBY: That moment was back in 2020. By then Navrina Singh had already clocked over two decades as a computer engineer in Big Tech, at companies like Qualcomm and Microsoft.

She had front seats to AI development. And she saw the writing on the wall about AI before a lot of us did. She knew that AI would transform our world.

She also knew that in order for this transformation to be a positive one, we would need to build responsible and safe AI. There’s too much at stake.

SINGH: Nobody was solving this problem. This was on the back of my mind for the past 10 years prior to that.

And I was like, if not now, we are going to be in a very messy world, where not only our democracy, our society, our planet are going to be beholden to this very powerful technological transformation and we as builders need to really own our responsibility. You don’t have to be really great to get started, but you have to start to be great.

EL KALIOUBY: So Navrina did what she always does: she took action. She founded a company called Credo AI that’s at the forefront of AI governance. They help their clients achieve responsible AI practices by doing things like ensuring compliance and adhering to AI regulation.

But what does AI governance actually mean? And why do we need it?

Navrina Singh has the answers. On this episode we talk about how we can build responsible AI through company and governmental policies. And if you work at a company that’s using third party AI software for research or hiring or really anything, this is a must listen.

I’m Rana el Kaliouby and this is Pioneers of AI, a podcast taking you behind the scenes of the AI revolution.

[THEME MUSIC]

Hi, Navrina. Thanks for coming on the show. So good to see you, how are you doing?

SINGH: I’m fantastic. Well, thank you so much for having me.

Copy LinkWhy AI governance must start before products ship

EL KALIOUBY: So today we’re going to talk about AI governance, and the way I think about AI governance is basically it’s a set of practices, frameworks, policies, tools that help us develop and deploy AI responsibly. But I wanted to hear kind of how do you think about AI governance, and why do we need it in the first place?

SINGH: Absolutely, Rana, what a critical topic for the time that we are living through right now, by the way. So I think about AI governance as really the competitive advantage that companies need to not only build trust in artificial intelligence, but also to win in artificial intelligence. And I think your definition is spot on because it’s not one thing.

It is really putting the right structures, whether it is accountability structures, policies, frameworks, tooling, to make sure that your AI investments are not only guarded with the right values, but you’re extracting the maximum ROI from these investments. I love giving this example. Think about artificial intelligence as one of your favorite fast cars, Bugatti or Ferrari.

SINGH: AI governance is truly the brakes and also the pit crew that makes these fast cars not only operationalize and move faster, but also making sure that you’re actually winning this race. So that’s what AI governance is right now.

EL KALIOUBY: A lot of the time I see companies think about governance, compliance, ethics after the fact. So you’ve built the product, you’re about to ship it, and then somebody says, wait, did we think about the ethics of all of this? Or is it safe? And it’s after the fact. And I guess one should really think about this from the get go.

SINGH: Absolutely, Rana. The companies and individuals who think about governance as an afterthought have honestly already lost in artificial intelligence because you can’t bring the same software building mentality to AI. And the reason for that is not only the scale and impact, but also how quickly that impact can go wrong with unintended consequences, if gone unchecked, and that’s why governance really needs to be front and center to your AI strategy. Especially as you start thinking about agentic AI, you’re going to have all these agents that are floating around representing you and what you need to be doing and the objectives you need to be accomplishing. If they’re not aligned to the right values and making sure they’re not governed, imagine what can go wrong.

Copy LinkHow Credo AI created a category before the market existed

EL KALIOUBY: Yeah, I want to come back to the AI agents angle, but before we go there, you founded Credo in 2020. That was really early in the kind of AI governance landscape. Nobody was talking about this beyond blogs. So what was the market and especially the investor response like?

SINGH: So first and foremost, there was no market in AI governance. We actually at Credo AI created and coined the term AI governance. Made sure that was socialized at the highest levels, not only in the private sector, but also in the policy sector. I currently sit on President Biden’s National AI Advisory Commission, and so really getting that opportunity to bring AI governance to the policy and government ecosystem, and then ensuring the analysts were brought into this category was, I would say, a big part of our focus in the early days.

EL KALIOUBY: Analysts. Specifically business analysts who use data to strategically understand a business including its processes, products and services. What do you mean by analysts being bought into this?

SINGH: When you have a technological transformation, many a times we start taking examples from history to apply to the current future. And what ends up happening is when artificial intelligence started to take off back in 2018, 2019, there was a thesis that, oh, governance is not going to be important.

That was first. And then the second was, if it is going to be important, we can use the same methodologies that we’ve used for other software.

And the challenge with that understanding and thinking, Rana, is artificial intelligence is a very dynamic technology. The static ways to govern cannot be applied to it.

In addition, the speed at which AI transformation is happening, and we’ve seen examples of that in the past 18 months, there are new emergent properties, especially with large language models, as well as multimodal systems that we have not seen before. So you can’t really apply patterns from the past into the current AI technology.

So we had to educate the analyst community because of the dynamic nature and the unknown problems in artificial intelligence. So there was a lot of education needed to create this category.

And then the past four and a half years have been really focused on not just talking about it, but actually operationalizing that through tools and products. And that’s what we are building and delivering to the world right now.

EL KALIOUBY: Credo AI customers have access to their AI governance platform. Think of it as a dashboard of all things responsible AI.

Customers can use this platform to get risk assessments on things like bias, discrimination, or even cybersecurity threats in their AI tools. They can also use the platform to reach policy compliance, and generate bespoke reports.

Copy LinkWhat good AI looks like and how companies define it

So I love what you said about how AI is constantly evolving and developing, whether it’s the algorithms or multimodality, which essentially is going beyond text to including computer vision and maybe voice and other sensors and other kinds of data, and even AI agents, like you were saying, technologies that are going to act on our behalf, take on roles that traditionally were done by humans and now go off and execute them on our behalf. We need a new framework and a new approach to governing these systems. Can you kind of go one level deeper? Like, what does that actually mean?

SINGH: Absolutely. Agency is a very powerful capability, and I would say one of the core characteristics humans have that we can reason and use our agency to our benefit. And one of the biggest things of making sure that that agency is guided in the right way is first through alignment.

So in governance, the first step is really aligning on what does good look like. And as you can imagine, that definition of good varies by individuals, by enterprises, by countries. So we are extensively focused on aligning an enterprise to what good looks like for them, aligning their technical and business stakeholders around what that good looks like, and then making sure that it is operationalized.

And as you can imagine, right now, in terms of alignment of AI systems, you can align to standards like NIST risk management framework. You can align to standards like ISO.

So the ISO standard basically defines what should a company do in terms of making sure it has the right oversight. As an example, ISO 42001 includes, do you have a responsible AI policy or a code of responsible AI policy within your company? Do you have a chief ethics officer that is responsible for all of responsible AI within the company, et cetera. You can align to regulations like the EU AI Act, or you can align to your own guardrails for facial recognition or speech recognition systems.

EL KALIOUBY: NIST risk management framework and ISO are government and non-government standards that help set guardrails for all kinds of industries.

The EU AI Act is a wide sweeping policy that will place guardrails on companies making and deploying AI. This doesn’t just effect European companies, but also companies, say based in the US, that provide services to Europe.

Like Navrina said, these guardrails can look like having an ethics team in your company or they can look like data transparency – so customers know how these models are being trained.

But for now what you need to know, is that these are some of the standards that Credo AI helps their customers meet.

SINGH: So there’s different mechanisms to align, but I think the most important thing to know is one size does not fit all. As a result, really aligning on what your AI systems should be doing and serving to your consumers is a very important step of governance. Now, once you’ve aligned on that, it’s really important to go and check whether your AI systems and your processes are doing what you’ve decided in that alignment.

And this is where a lot of interrogation and testing and evaluations of your data sets, models, and use cases come into picture.

EL KALIOUBY: Okay, so companies need to be thinking about how they’re aligning with industry standards. They also need to be establishing their own internal guidelines.

On the ground, this translates into practical decisions—like choosing less-biased AI hiring tools or evaluating how a generative AI model manages private company data. We’ll get to that and more in a minute after a short break. Stay with us.

[AD BREAK]

Copy LinkTurning governance principles into real decisions inside a business

Before the break, we defined AI governance and discussed some of the guardrails companies need to be thinking about when it comes to AI. Now, let’s dive into some concrete examples of what this actually looks like.

So I want to give a few examples. At the production company that produces this podcast, we actually have a listener’s bill of rights in the age of AI. And one aspect of this bill of rights is that anytime we use AI as a voice, or even if you’re hearing or seeing an AI clone of myself, for example, we are basically going to disclose that this is an AI versus a real person.

So that’s one example. And then at Affectiva, the company I co founded, we were developing all these computer vision algorithms, say a smile detector or a drowsiness classifier. We were just really committed to building this ethically and mitigating any data or algorithmic bias. So these are kind of two examples in my universe.

Can you give an example of what AI governance and building this responsibly and safely would look like at, say, a bank.

SINGH: Yes, absolutely. So one classic example, which applies irrespective of the industry, is everyone’s hiring, right? When you’re hiring, a lot of companies are dependent on third party tools, and some of these third party tools can come from, Eightfold AI, Workday, name it. For something as classic as that, you can imagine that you as an organization need to decide how are you going to make sure that these systems are actually accepting a diverse set of candidates that aligns with your company policies and might not be unintentionally leaving out a certain demographic just because they have not been trained on those demographics.

So if you are an organization based in New York, New York actually passed a law about a year and a half back, which is called New York City law number 144, that requires organizations that are buying third party AI systems to do a fairness audit. So a very classic example that we are seeing across all enterprises, especially if you’re buying a third party HR system which is based on machine learning, is you have to do a fairness audit.

And so this is where governance really comes into place because you want a standardized way that all your teams are actually doing that analysis at scale.

EL KALIOUBY: So say I’m a big organization and I’m using one of these AI tools for hiring. Would I then engage with Credo AI to help ensure that these tools are fair and kind of meet these regulations?

SINGH: Yes. Since we are in this interesting world of generative AI, a lot of our enterprise customers are making massively big bets in AI, and as you can imagine, us included, we are seeing really amazing productivity gains from using generative AI tools, whether it is for coding, whether it is for marketing, customer support, and as you can imagine, as a startup that is really valuable.

But one of the challenges that happens when you are bringing in these generative AI systems into your organization is again, unintentionally you are not thinking about the risks.

So if there is a Gen AI system and you are providing them your own proprietary information, how are you making sure that they’re not training on your data that you are providing? Or second, you can think about data retention policies. I, as a company might have a data retention policy that might be zero days, but maybe some of these Gen AI companies have 30 days, 60 days, et cetera.

When an organization is buying third party gen AI systems, we do a couple of things right out of the box.

We provide you risk profiles of many of the systems that our customers are using, and you can actually publicly see them on our website. And two, ongoing governance really depends upon the context of use. When you’re using these large language models or multimodal systems for marketing versus search versus coding, the context of use matters and the context is what we use to define the guardrails.

And so we show you within that context how your systems are performing and whether they’re not compliant.

Copy LinkWhat CEOs should do first to get AI under control

EL KALIOUBY: So I’m a member of the Young Presidents Organization. It’s a YPO. It’s a global network of about 40,000 CEOs globally. And just over the past year and a half, everybody’s freaking out. All these CEOs are freaking out, trying to figure out how to incorporate AI into their businesses and do that in a safe way. What’s your advice for these CEOs? Like how should they think about governance?

SINGH: Yeah, first, before I go into the advice, I’m glad they’re freaking out, but I also would advise that they should not freak out because this is such a great opportunity. We are living through such an amazing time in artificial intelligence that we have one of the most powerful technologies at our hands that we should be using to figure out how we are going to show up as leaders in this age of AI.

So a couple of key things. The first and foremost is, if you ask these enterprise leaders where true artificial intelligence and gen AI is actually being used in their organization, most of them won’t know. So taking stock of where AI and machine learning initiatives are within your organization is literally the first step.

EL KALIOUBY: And also knowing where your data is, right? Kind of taking stock of your data. Most companies have no idea, or rather it’s disparate, it’s all over the place. Yeah.

SINGH: In governance, one of the shifts that we are seeing is absolutely that governance requires you to have good data hygiene, because good data means good AI. But a lot of business impact is dependent on the actual application — the context matters, as I’ve been mentioning. So really understanding where AI is used — you might have data sitting anywhere, but if you’re not putting that to work, you honestly don’t have much risk. But when you start putting that data to work through AI applications, that’s what you should be paying a lot more attention to.

So the first thing is really taking stock of where you are actively using AI applications within your organization, whether that is internal — let’s say for hiring, for marketing, benefits, et cetera — or whether it is for external product creation. Once you’ve taken stock, the second thing is really aligning across your leadership on what does that good look like.

Because AI governance is not a grassroots initiative. It needs to come from the top. And then the third thing is once you’ve aligned on what good looks like, that’s when you start operationalizing and enforcing the standardized way to have that accountability across all your AI implementation.

Copy LinkWhy companies resist governance and why that mindset is risky

EL KALIOUBY: I love that you said how important it is that this starts from the top. I always say the chief executive officer, the CEO, is also the chief ethics officer. It has to be a part of the values of a company. I’m curious, what are some of the biggest challenges you’ve faced convincing companies to invest in AI governance?

SINGH: I would say it’s the same across even the investor base. It’s a very interesting dynamic — everyone will raise their hand for innovation, but when you start talking about who wants to do it responsibly, everyone just lowers their hands.

And then when you ask who is actually going to be held accountable for what you’re putting out in the world through AI, everyone disappears. So I would say that we are going through a very similar motion in artificial intelligence where the excitement around innovation is all time high.

But the actual implementation of doing it responsibly and safely and understanding the risk is not there. So some of the resistance I would say is first, they’re like, it’s not a priority for me right now.

There’s no regulations. And I think that’s what we need to change very quickly because you don’t do AI governance to meet a checkbox, which is regulation. You are doing AI governance to build that competitive advantage and to lead with trust because you’re leading through transparency with your customers.

And then the second thing that we hear quite a lot is it does not apply to me. And I think that’s where it’s like, it does apply to you. Let’s start taking stock of where artificial intelligence is being used with or without you knowing.

Because in big enterprises, we have a lot of shadow AI use cases that employees — they all are using AI tools, and I would say that if you’re not using AI tools, you’re already behind. So we encourage everyone to use these AI tools. But again, how do you do so with education and with responsibility and with governance tools like Credo AI?

EL KALIOUBY: If you work at a company that uses or even makes AI tools, the gears are probably turning right now. If you haven’t yet started your AI governance journey, now is a GREAT time to do so!

But AI goes beyond individual company responsibility. What is our government’s role when it comes to AI regulation? We’ll get to that after a short break.

[AD BREAK]

Copy LinkHow businesses can keep up with fast moving AI regulation

So let’s switch gears to the regulatory landscape. There’s so much happening on this front, and you help companies engage with how to use AI safely, but do you also hold their hands and help them navigate this regulatory landscape?

SINGH: Absolutely, Rana. Just like artificial intelligence is a new technology, the policy ecosystem is trying to evolve as fast as they can with this technology as well.

We take on the burden for you to keep track of all the emerging regulation, standards and best practices, and then guide you through making sure that you can very responsibly apply that to your AI applications. We’ve been very focused from day one on not only providing the tools that operationalize the policies and regulations and standards, but also bringing AI expertise to policymakers so that whatever they are proposing actually makes sense in these operations pipelines.

EL KALIOUBY: So last fall, I was in Belgium for a few weeks on my Eisenhower Fellowship and I met with several of the EU AI Act legislators, which was really fascinating. What should we all know about the AI Act? And can you kind of recap the timeline for implementing this regulation?

SINGH: Yeah. So a little bit of historical context — the European Commission started working on this almost five years ago.

EL KALIOUBY: Yeah, which people don’t realize, right? Yeah.

SINGH: So I really say kudos to them for having put the amount of thought and effort into bringing multiple stakeholders together to create the EU AI Act as a first piece of legislation.

I think what has been really exciting to see with the EU AI Act is they’ve brought in a very comprehensive risk-based approach and rights-based approach. So rights are human rights.

So they are really thinking about when these AI applications are going to be used in Europe, how are they going to ensure that it actually works for the European citizens. So marrying the rights-based approach with the risk-based approach has been really exciting to see for the EU AI Act. The core things everyone should know is the EU AI Act has certain risk categorizations depending upon whether you are a developer of artificial intelligence systems, or a provider of artificial intelligence systems. And then based on where you fall in the risk profile, there are separate requirements that apply to you in terms of quality management, risk management, and requirements for those applications.

And then obviously the key idea, without going into too much detail, is how are you making sure that oversight is happening throughout your AI lifecycle, and that you have the right governance and audit artifacts that can then be validated by conformity assessment mechanisms, along with alignment to standards that Europe is working on, which are going to — the timeline for that is end of next year.

So the idea behind the EU AI Act is really thinking about the risk of your applications and then making sure that across those risk profiles, especially for critical risk and high risk applications, you’re putting the right hygiene in place to do risk management as well as quality management.

EL KALIOUBY: So an example of a high risk application would be biometrics surveillance, for example.

SINGH: Yeah. So if you think about facial recognition use in public spaces, that’s such a great example, Rana, because what does fairness mean when you think about facial recognition systems? There is no common definition. There are 21 definitions — happy to share the paper with you that’s been floating around for multiple years now — but again, what does good look like in terms of what is a good facial recognition system?

But it is high risk. And then you basically go and do certain testing on your data sets, your models, and your application to make sure within the context of that facial recognition system, you’ve managed all the risks.

EL KALIOUBY: We’ve talked about some of the risks of facial recognition technology on the podcast before. My main concern is the risk of data and algorithmic bias. On a small scale this can mean your phone camera not identifying your face.

On a larger scale, this could mean a person getting arrested for a crime they didn’t commit due to faulty facial recognition.

Copy LinkWhat governments should prioritize beyond compliance and enforcement

You also serve on the federal committee that advises the Oval Office on AI. What’s being discussed at these meetings and what should we all expect?

SINGH: It’s just been such an honor the past two years to be on the National AI Advisory Committee that is part of the Department of Commerce. What is being discussed is really thinking through the transformational nature of artificial intelligence, everything from AI literacy. Rana, if you start thinking about users who are, with or without their knowing, using artificial intelligence — the majority of them don’t even know the basics of AI. So the question is how can we as a country build capacity around AI education, which is a very important and critical topic.

EL KALIOUBY: Love that.

SINGH: A lot of time is spent thinking about that.

And I know your kids obviously have grown up around you embracing these technologies, but not everyone’s kids are. So how do we think about K through 12 education? That’s a big topic for us. We think very deeply about evaluations and benchmarks, and I am thrilled for the work that the National Institute of Science and Technology and the US AI Safety Institute, which was recently established, is doing in that space. Because again, what are you measuring around these systems and what does that good look like in making sure that these very powerful foundation models actually start serving us as they show up in national security, as they show up in the democratic process for elections, as they show up in all the misinformation that we’ve been talking about recently. So how do we make sure we have the right evaluations and benchmarks and standards?

That’s a second very important topic, and then the third topic which I am also not only leading but very excited about is how we work with our international allies. Because as the United States, we are leading in this age of AI, we are at the front end of all the scientific and technological advances in AI, but it’s really important for us to bring our allies together. So what does that international cooperation and harmonization of standards look like? That is a really important topic that we are paying a lot of attention to.

Copy LinkWhy deepfakes and misinformation require shared accountability

EL KALIOUBY: There’s a particular issue that a lot of people are thinking about when it comes to AI regulation. This past election, we saw a slew of AI generated content about various candidates. We also saw deepfakes that spread disinformation.

Some people are pushing for regulation around AI generated content, but Navrina thinks that it shouldn’t stop there.

SINGH: It’s very hard, Rana, and this is where we need to have shared accountability. So obviously through tooling we can do a little bit. We’ve been discussing the concept of watermarking, where essentially you are embedding some information into AI generated content that allows you to decipher that this is AI generated and not human created.

But that’s just one piece. I would say this is the moment in time, but beyond tooling, there’s a lot of responsibility that individuals as well as enterprises need to undertake to make sure that these misuses are managed and, more importantly, completely eradicated. It’s a very hard problem. I wish I had a solution for it because it’s not one thing. So what is our shared responsibility — regulatory, non-regulatory — that needs to come into this ecosystem is a hard one, but we need everyone to come and provide solutions around it.

EL KALIOUBY: Yeah. So I’m a huge advocate of regulation and thoughtful regulation, but I also see a challenge in that the pace at which AI innovation is happening, the pace at which AI is being created, deployed, and used — it’s so hard for governments and regulators to keep up. So what’s the solution here?

SINGH: Yeah. So I’m an eternal optimist, otherwise I wouldn’t be a founder. So a couple of things that I’ve been advocating pretty extensively in the past two to three years: first and foremost, private public partnership. This is the moment in time where we need to go and inform policymakers what is realistically possible as guardrails and what is not possible as guardrails.

And then on the flip side, we need to not only take these policies but put them in action and make sure that we are holding private companies, especially big tech, accountable to these guardrails. Because as you and I have seen, and you’ve been in the space for a long time, voluntary commitments don’t work.

The second thing, which is really critical, and this is something that we are trying out at the state and local level, not so much at the federal level, is adaptive policymaking.

EL KALIOUBY: Adaptive policymaking. Interesting.

SINGH: And Rana, the interesting thing is this is where we need to give regulators the permission to move fast, and sometimes it’s not going to be perfect policy, it’s not going to be perfect regulation. But how quickly can we iterate on what we’ve learned from the market and be able to modify and adapt it?

But right now I think we are holding policymakers to a very different high standard than we hold technologists to, right? As a technologist, you know our software release process. We roll something out, doesn’t work — okay, great, we’ll roll out another version. Let’s just iterate and figure out what works.

We need to be okay with that in policy. And I think it’s obviously very hard to implement, but the concept of adaptive policymaking is something that I’ve been socializing and testing out at the state and local level to really understand when we put something out there within a sandbox, see how it gets implemented.

If it’s not getting implemented well within that sandbox, let’s iterate on that process till we are able to put a policy out that actually works.

Copy LinkHow AI could eventually help govern other AI systems

EL KALIOUBY: Love the concept of adaptive policymaking. I’ll be watching out for that. So, to wrap things up, if you could have AI do anything for you, what would you want it to do?

SINGH: Professionally, I would love to see AI actually govern other AI systems. So something that we’ve been thinking deeply about is how can we leverage large language models to actually create better governance mechanisms?

And then personally, what I would love AI to do is literally take over all my scheduling as well as travel plans, because I’m a horrible planner. And so for me to really have a system that can just plan and schedule everything on my travel would be such a welcome relief, but it hasn’t happened yet. I haven’t found an app. If you have recommendations, let me know.

EL KALIOUBY: Yeah, I’ll let you know. It’s definitely something I’m looking out for as well. Well, thank you so much for joining us on the show today and talking all things AI governance.

SINGH: Well, thank you so much for having me, Rana.

EL KALIOUBY: If there’s one thing that you take away from this conversation, I hope that it’s how critical it is to start your AI governance journey. You don’t need to be working at an AI-forward company to do this. The reality is, a lot of us are already using AI in the various industries we work in.

And look, I know that we say on this podcast that if you’re not using AI, you’re already behind. I stand by it, but I also think we all deserve some grace. I’ve been working in the AI space for over 20 years. But for a lot of us, this is still very new. So it’s OK if you don’t have all of your ducks in a row right now. What’s important is that when you start your AI journey, you keep AI safety and responsibility top of mind.

So have you started your AI governance journey? Are you inspired to?