The making of an AI clone

Note: Transcripts are automatically generated from episode audio, and are not fully corrected for spelling, grammar, and formatting.

RANA EL KALIOUBY: Have you ever wished that you could be in two places at the same time? Or that maybe there could be a second you – some carbon copy version to finally get through your to-do list?

Well in the near future something like this can be possible. I’m talking about AI clones – a digital version of ourselves that sounds, looks, and thinks like us.

For the past several months, I’ve been developing different AI versions of myself using a couple of the big platforms out there. And I got to share the experience on stage with one of the experts in this emerging field of virtual humans, Natalie Monbiot.

Today we’re sharing that conversation with you – live from the On Air Fest stage in Brooklyn, New York.

I’m Rana el Kaliouby and this is Pioneers of AI – A podcast taking you behind the scenes of the AI revolution.

[THEME MUSIC]

EL KALIOUBY: Hi everyone, this is Rana Alkayoubi and my guest today, Natalie Monpio, and we have an awesome show for you today. I want to kick us off with a video, so let’s watch. Welcome to On. OK – so at this point during the live taping, we played a video demo of one of my AI twins. You can watch it by heading over to our YouTube channel. But here’s the audio version.

Welcome to OnAirFest. I am so excited to be here with you. Today we’re going to talk about AI clones. Let’s get started.

Right, so that was cool. Kind of me, I guess. I love that I am speaking Spanish and Mandarin, which I do not speak. So I just quick show of hands. If you speak Spanish and Mandarin, show of hands — did that sound kind of accurate-ish? Okay. A lot of ish. Okay.

NATALIE MONBIOT: Spanish, yes?

EL KALIOUBY: Yes. Okay. All right. Great. Well, I can kind of judge my facial expressions. They kind of look like me, but something’s a little off, and the voice doesn’t exactly capture.

I like to say my soul. But that was my AI clone. That was my digital twin. And I’ve been experimenting with all sorts of AI cloning technology. And today we are going to go behind the scenes to uncover what an AI clone is, what are some of the applications, and of course what are some of the ethical considerations.

But before we get into all of that, I just want to take a moment and introduce the two of us. I’m Rana Elkayoubi. I’m an AI scientist, entrepreneur, and now investor, and also the host of the podcast Pioneers of AI. And my guest today, Natalie Monpiot, is a dear friend, and she is in my opinion the world expert in the virtual human economy. Natalie, do you want to introduce yourself real quick?

MONBIOT: Sure thing. So just to explain the virtual human economy, maybe I am the world expert because I coined that phrase. So not much competition so far. I define the virtual human economy as a new era in which we each have AI twins that we can put to work on our behalf to lead better lives.

And in terms of where I got that from and where my inspiration was, I’ve been a long time emerging media strategist. Then about six years ago, I joined a fledgling AI avatar startup, Hour One, and helped build that business from the ground up back in the day when AI was basically only known as deepfake.

So kind of getting us out of the rut of the deepfake era into kind of positive use cases of AI twins and avatars and different commercial use cases. And through that journey, I got really excited about this idea that behind every AI avatar or any AI twin, there can be a human being behind it that can benefit from this technology. So to me, AI twin technology is an enabler for human beings in the era of AI. And it’s a way that humans can collaborate with AI in a way that they can win.

Copy LinkWhat AI clones really are and why the next question is now what

EL KALIOUBY: So define what an AI clone or an AI twin is very simply. And what did you think of mine? We’ve known each other for over a decade now. Did it sound or look like me?

MONBIOT: I mean, to an extent, I’ve seen so many versions of AI avatars and AI twins. I think there’s a certain level of expectation and expectation gap. It’s pretty good, right? At delivering to the camera a functional message, especially when it’s doing things that you can’t do.

I think there’s this kind of a hum. And whenever I saw my AI avatar speaking languages I didn’t know, I’d be like, wow, actually — do you speak that language? Like I kind of started to believe it. And so it was kind of this weird feeling. It’s like, no, but actually it’s also true because in a way I do speak those languages, so it’s starting to kind of feel that empowerment through AI, but yes, there’s still definitely a gap.

And I think the visual aspect of an avatar is very ambitious to actually communicate, as you say, doesn’t represent your soul. And is that really what we should be trying to do anyway? Are there actually more functional ways that we can use our AI twins to actually benefit us?

I would say something that people come to me quite a lot, especially towards the end of last year, is: I created my AI avatar, I cloned my voice — but now what? How do I profit from this? How do I benefit from this? And I think there’s been a bit of a bubble in that respect, where people haven’t really reaped the benefits from their AI clone. So I think we’re transitioning into an era now where people have gotten over the fact that you can do it, and it’s kind of like, well, now what? And so I think we’re entering a really interesting phase of being able to clone aspects of ourselves that are more valuable and that we can use to profit.

Copy LinkHow video clones are built from a few minutes of training data

EL KALIOUBY: So let’s go behind the scenes and kind of explore what it takes to make an AI clone.

And I want to start with products that mimic your likeness. In this case, it was my visual appearance and my voice. There are so many products out there, and to be honest, they’re all still evolving and none of them are perfect yet. My video clone — I used a company called HeyGen. And the process was actually pretty simple. I logged on, signed up on their website, turned my video on on my laptop, created a three minute video, just kind of pontificating for three minutes, uploaded it, and then basically I am now able to type in whatever script in whatever language and my clone will present it on my behalf. So you’ve been in this space for many years. What is the technology behind this kind of video cloning?

MONBIOT: Right, so what we’re essentially doing is we’re taking a little bit of video footage — in your case, just a couple of minutes. In some cases, it’s just a photo. Maybe if you’re following the AI news stream or fire hose, you’ll see different breakthroughs from different companies from all over the world showing what kind of lifelike videos you can create, even from a photo. So essentially what it’s doing is it’s taking your characteristics, the data points in the training data, which is essentially whatever you gave the AI to train on. And it is replicating that and you breathe life into this replication of you. And then it mimics how you might have said those words yourself.

EL KALIOUBY: And what is also really cool is it does a lot of lip syncing. How does it do that?

MONBIOT: Well, that’s kind of the magic of the AI algorithms and something that our category of AI avatars has been extremely focused on for the last few years, because it’s actually very difficult to nail. Just a little tidbit: at the very beginning of Hour One, so 2019, we actually used a different technique. We actually had a human being behind the AI avatar, moving the AI avatar kind of in real time, and the lifelikeness and expressiveness was much more accurate and compelling because actually you had a human being kind of driving it. But that was not scalable. So we actually had to take a big step back in terms of the effectiveness of the experience in order to be able to build a product that actually was more scalable.

EL KALIOUBY: That’s kind of interesting, because that’s like motion capture in the movie production space, right? Where you have actually a person driving the actions, driving the movements, and the quality of the output depends on that human being. Which is a great segue to the next company we want to feature. It’s another big company in the space called Respeaker, and they make voice clones.

It’s actually the company behind the movie The Brutalist, where Adrian Brody spoke perfect Hungarian, thanks to AI. They basically used Respeaker to have him speak Hungarian and we’ll show a demo in a minute. But Natalie, how does that technology work?

MONBIOT: Right, so it’s actually a little bit like what we were just talking about. There’s a real person’s voice that’s driving your voice. And so it’s giving it the emotion that is desired, but with the mask of the voice that you want it to actually be communicated in.

And so similar to the earlier technique that I mentioned at the beginning of Hour One, it actually creates a much more lifelike effect, but it isn’t scalable, which is perfect in some use cases. And in others, it’s not, when scale is the objective.

EL KALIOUBY: After a short break, we will hear what this technology sounds like in action. Stay with us.

[AD BREAK]

Copy LinkWhy voice cloning feels more lifelike and where it saves time

EL KALIOUBY: We’re going to talk about the use cases of some of this in a second, but let’s listen to a demo of Respeaker. So the way this worked is essentially we had our producer, Rachel, record a promo script in her voice. So let’s listen to that.

So this is Rachel’s voice and she recorded the script. So now let’s listen to how Respeaker transforms her voice into my voice. And to train this model, we used excerpts from episodes of Pioneers of AI. So my voice as a podcaster, if you like.

We trained a different version of this using data from my audiobook.

I recorded my book. That was a whole week worth of data. I think we only used a short clip, so let’s listen to that.

Natalie, what do you —

MONBIOT: I like that one.

EL KALIOUBY: Like that one?

MONBIOT: That one a lot. Yeah. That really sounds like you. Yeah, no. That’s great. It’s like your intonation, but it’s Rachel’s.

EL KALIOUBY: Yeah, exactly. It’s incredible. Yeah, well Rachel has gotten to know me really well, so I think she’s also channeling a bit of me. What I find interesting about the first one is it’s a little like news anchor Rana kind of thing. And sometimes I’m like, ah, maybe I should sound like these people, these alternative versions of Rana. I still think it’s missing some of my energy though. I don’t know how we capture that.

MONBIOT: I think we should leave that part to you. We should delegate our AI twins to have very specific roles and to execute specific functions, and it’s your job to use that free time to kind of enhance your soul.

Copy LinkHow to turn your voice and likeness into scalable business value

EL KALIOUBY: Okay, I love that. I’m game, I’m down for that. Let’s talk about some of the use cases. One of the applications we’re exploring in our podcast is using ReSpeecher and my voice clone for voiceover scripting. So, instead of me carving out time every week to do some of the voiceovers, we can just have Rachel record it and ReSpeecher will reformulate it in my voice clone.

Which would be a huge time saver. And I think it makes our team more productive and efficient. I think a lot of people can easily see the use cases in entertainment and media, like movie production, podcast production. What are some of the other use cases that you see for these types of technologies and clones?

MONBIOT: Talking about voice in particular or more broadly? Video. Okay. So I’ll just give you an example, because I’m always thinking about how can we actually use these technologies to benefit and quite literally profit. Is everyone here familiar with ElevenLabs? So ElevenLabs allows you to clone your voice or access millions of cloned voices and generate endless amounts of content with these clone voices. It also allows you to do so with your own voice and you can actually put it in the ElevenLabs marketplace.

And you can make it available for use, and you can make money. So actually, sort of every month, I get some random thing in my account. Like 20 bucks, 30 bucks, and I’m just making money by making my voice available for commercial projects.

EL KALIOUBY: Who’s using your —

MONBIOT: I actually don’t know, and maybe I should.

It’s like on the bottom of my to-do list to kind of work that out or sort of manage that. But you can do things like set your price, and there’s no commitment on my part in terms of how long I would make my voice available. So I could just remove my voice any day.

The value of my voice is lower, but I can increase the value of my voice by setting parameters like, I’ll commit for a year, and so therefore companies will invest in your voice and pay more for it. So it’s kind of interesting to think about separating out different aspects of you that you can clone and put to work on your behalf.

And then I think it comes back to what aspects of you are valuable that you want to put to work on your behalf, and there are an increasing number of ways that you can kind of separate yourself out.

EL KALIOUBY: Yeah, it’s the British accent.

MONBIOT: Not really doing it for me on the 20 bucks a month thing, though. Yeah, I’ve got to work on that.

EL KALIOUBY: But do you have any idea — do you think you’re reading an advertisement somewhere for some brand?

MONBIOT: Yes, I think probably. And again, I should probably look into it. But in the terms of use, as with all these AI avatar companies — so at Hour One, the thing that really inspired this idea that any human can create their AI twin and profit from it came from the fact that behind every avatar that we had, we had cloned thousands of people, many of whom made their AI twins available on the platform and kind of made money passively while they slept. And governing these contracts are terms of use.

Such things as it’s not so much that you can select exactly where you want your likeness to show up, whether that’s you in video or in voice. It’s more about where you’re not going to show up, right? So it’s like: no sexually explicit content, nothing illegal, no gambling, no political content, etc. So those would be the platform parameters, and you can read those terms of use and see if you feel comfortable with that.

Copy LinkWhat enterprise adoption reveals about the strongest real world use cases

EL KALIOUBY: Can you talk about the example with Berlitz? Because that’s also an amazing use case.

MONBIOT: So actually this is what we want. This is our first ever enterprise customer that probably nearly broke us, but in the end actually propelled us forward, which is often what happens in startups. So this is back in 2020, we got approached by Berlitz, which is a 150 year old language school, Japanese owned — not what you would think is going to be the most innovative company, the first to use AI avatars and generative AI, or synthetic media as it was called back then.

But they had an absolute need to digitally transform, scale that classroom based learning into a platform and product that people could access all over the world for a lower price point. And they tried filming their instructors the usual way.

But tens of thousands of videos needed to be captured to have all these different clips in different languages and they couldn’t manage doing that with a human workforce. It became completely untenable — not just the volume and cost, but actually also just the management. You know, the takes, and then it’s like, oh, we need to redo that — bring them back the next day, they’re wearing something slightly different. It just was unmanageable.

So they looked for — it was almost like the last resort, but the only one — which was to use AI instructors, some of whom were based on their actual instructors and others were based on actors. They were able to fulfill their diversity objectives as well as a global brand by introducing AI instructors that could teach multiple languages that the actual instructor in real life wouldn’t be able to teach all of.

That’s the kind of thing that got me really excited. Like, when can you create outsized value and deliver something that you couldn’t possibly do before because of this technology and create a better customer experience as a result?

EL KALIOUBY: We need to take a short break. But more of my live conversation with Natalie in a minute.

[AD BREAK]

Copy LinkWhy cloning expertise may matter more than cloning appearance

EL KALIOUBY: So let’s move into a different type of AI clone. So far we’ve looked at visual and auditory clones, but one area that I know we’re both super excited about is how do we capture our knowledge, our brain trust, and what does that look like? Wisely is one of the companies we are going to feature next, and you are currently working quite closely with them. So tell us a bit about their vision and what are they trying to achieve?

MONBIOT: Yeah, absolutely. So what they’re doing is, as I mentioned earlier, transitioning from this period where we’re sort of like, okay, this technology exists, but now what — how do we actually put it to work on our behalf? And what are the most valuable aspects of us that we can actually scale and augment?

And for many people, that’s not how they look or sound. That’s actually their body of work or their expertise. And so what Wisely is doing is they’re building a verticalized platform for experts to be able to clone their expertise and manage it in an AI platform, and to then make that expertise available to people anytime.

And so the benefit to the human expert is that their time is freed. They can have their AI twin trained on their knowledge interact with potential customers, and also use AI in the background as an assistant to help actually manage all of their content — have the AI brain trained on the content.

A firehose of your own data. So connect it to LinkedIn, connect it to Notion, connect it to Google Docs and have it constantly be updated with your expertise. So this thing is constantly becoming more and more useful.

EL KALIOUBY: And I guess for the rest of us, it democratizes access to expertise. It’s like having Andrew Huberman in all of our pockets kind of thing.

MONBIOT: Right, and actually relating back to the Berlitz case study, I think a really interesting future use case is making your expertise available to more people at a lower price point and in different languages that you don’t speak. So imagine different markets where an individual can’t usually afford to access that expertise, and to now be able to not just afford it, but also have it communicated in their native language. So I think the opportunities are enormous.

EL KALIOUBY: It’s very powerful, very powerful. So I signed up to be a guinea pig with Wisely, and we created an AI clone of me. Do you want to talk a little bit about what kind of data we used to train my Wisely clone? Absolutely. So as we mentioned before, we’re not trying to clone Rana’s multifaceted person that’s done a number of different things. So for this experiment, we’re like, well, why don’t we take a big body of data that already exists, which is Rana’s memoir. And why don’t we train her AI twin on that memoir — the memoir tracks her experience as a founder.

MONBIOT: The founder of emotion AI and the startup Affectiva. So we trained her AI twin on that data. In addition to that, a bunch of interviews, because we’ve got that data because a lot of that book promotion was done virtually during COVID. So there’s so much data. And so we’re able to add that data into Rana’s AI twin and train it just on that. And then actually, as of tomorrow, the platform will be voice enabled. But for this audience and for today, we’ve actually been able to demo AI Rana with voice.

EL KALIOUBY: So let’s give it a try. And this is, by the way, a pre-recorded video of our producer interacting with my Wisely. So you’ll hear Rachel first, and then my Wisely, my voice will respond.

Rana, You are a pioneer in emotion AI. Can you define what emotion AI is and tell me why it matters?

EL KALIOUBY: Yeah, it’s thinking. It’s thinking.

MONBIOT: This is some natural latency. It’s prelaunch.

It’s basically a subset of artificial intelligence that focuses on recognizing, understanding, and responding to human emotions.

And they play a huge role in our decision making processes and interactions with others. By integrating emotional intelligence into AI systems, we can create technologies that are more empathetic and responsive to our needs. What do you think? Are you excited about the potential of emotion AI to transform the way we interact with technology?

EL KALIOUBY: Natalie, what do you think? What do you think of my Wisely?

MONBIOT: What do you think?

EL KALIOUBY: It sounds like the content is definitely how I would answer this question. It sounds very authoritative though, and I feel like I need to become more like that.

MONBIOT: Authoritative. I know, it’s funny when you kind of aspire to be like your AI twin. It sets a new bar.

Copy LinkHow consent control and trust shape responsible AI twins

EL KALIOUBY: I will say, and this is a great transition to a conversation around the ethics and the ethical considerations of all of that. It took us about two or three weeks for our team to get comfortable with the terms and conditions, because I’m giving up my IP, right? I’m giving up my knowledge, my — stuff I wrote, my book. And I think that these are some of the things that we have to grapple with as we create these clones. So let’s talk about the ethical considerations. The first one, I guess, is control and consent. What does that look like? What are you seeing companies do? And how do we get consumers comfortable to share their expertise in a way where, perhaps — because there could be bad actors, this could go really wrong.

MONBIOT: So I think it’s really interesting that we did have a lot of back and forth and were very thoughtful about this process.

But I think where we got comfortable is that you’re actually not giving up your expertise. And so the whole premise and the parameters are that you are only training your AI twin for your use with your data. And so talking about consent and control — this is something that was very focused on at Hour One as well — when you’re creating your likeness, who has access to this? And do you have the right to be forgotten? Can you remove your data? Like all of these things are really important. So we went through all the fine print to make sure that you were protected.

And I think a lot of it is also emotional, right? Because in a way, this is personal. This is your book. This is your baby. You put so much work into it. What’s happening to it when it’s being trained with AI? So a lot of these things are kind of like: how do you feel about this personally? There’s a little bit of an emotional hurdle and certain barriers that you may or may not want to cross on a personal basis.

But I think in the end, what really matters is whether you feel like you are benefiting from doing this. So in general, whether you’re creating an AI twin for yourself or creating AI twins or avatars for your customers, are you creating outsized value — value that couldn’t have existed otherwise?

So for example, I think bad practice as a business would be to replace people interactions that people love with an avatar, right? That’s not a good customer experience. But if you can make your customer experience or customer service more available than it was before through AI, then maybe your customers would actually really appreciate that. Or make it available in different languages — your customers would really appreciate that. And in your case, is the value that you’re going to get for yourself enough to make you trust this process? Is this twin going to help you in what you’re trying to do? So I think it always comes back to that. And yes, we just talked about consent and control. Consent is not really subjective, and control is a bit subjective, but really understanding it — I think it’s.

EL KALIOUBY: And how much autonomy do you want to give this AI clone, this AI agent? At some point it might have agentic abilities as well. The Wisely AI clone of me is interactive, right? And I want to make sure that — for example, one of the use cases we were kind of thinking about is, with my investor hat on, we see hundreds of companies come our way, and there is no way I would be able to have a one-on-one conversation with every founder. However, if my AI Wisely clone knows how I think, and it actually has a rubric for how we evaluate these startups, potentially it could have a quick 10 minute conversation with every single company that comes our way.

That’s way better than responding with a one liner, or even worse, ghosting these companies, which a lot of investors unfortunately have to do. But I wouldn’t want it to, — I would have to trust it if it’s going to actually talk to people out there on my behalf. And so trust is a big consideration.

MONBIOT: I think it is absolutely the critical consideration. If you don’t trust it, you’re not going to use it. And other people are not going to want to use it. And it’s actually going to erode trust — that would be the worst thing that it could do. And so I guess it’s like.

I actually think that’s something you could do today with Wisely — like that exact thing. And so you can try that in a contained thing where it’s not outward facing, test it, see your comfort level and such. But I think what’s also interesting — and this maybe is where you would have to get really comfortable with that first before going where things are going — is then to train it on what is called a small language model.

Okay. Which is not like a large language model, which is just all data everywhere that’s publicly available, but specialized data. So if Rana wants to take her AI twin that is conversing with all of these founders in this giant list that she would never be able to get to in real life, at some point, would she want it to actually be a little bit more proactive and make recommendations or decisions or actually book a call with the real Rana?

EL KALIOUBY: If a company’s exciting and meets our criteria kind of thing. Exactly.

MONBIOT: And the criteria can also be market data, subscriber data, financial data. So you can start to enrich your AI twin. And I think that’s really interesting — we’re talking about a scale use case, right? Like you don’t have time. But then you also don’t have all of this real-time market data in your head at the same time. So how can you actually augment yourself with AI? And I think that is the future.

Copy LinkWhy disclosure authenticity and human meaning matter most

EL KALIOUBY: At some point people will just want to talk to my clone and not me. What about disclosure? How important is that? We have a policy at Wait What, the production company of Pioneers of AI, that we disclose when AI is being used. Is that common? Is that uncommon?

MONBIOT: I think it’s absolutely critical. And I think that in most good commercial use cases, it happens. But I think, first of all, if you think there’s any benefit in not disclosing that you’re using an AI twin, then just stop, right? There’s no benefit in that.

You should be only using an AI twin if you’re creating value that couldn’t have existed before. And because you’re doing that and you can only do that with AI, shout it from the rooftops.

EL KALIOUBY: Absolutely. What about authenticity? When should one use their AI clone versus not?

MONBIOT: Yeah. So I think the question of authenticity is so interesting in this era, because I think there’s authenticity within the use of your AI clone. And then there is the authenticity of real human interactions, which I think become even more prized.

So the thing that’s not scalable is real in-person, live attention, right? This is not scalable. I wrote a post about this and had all these people jumping in. But with AI agents, you can scale attention. It’s like, no, no — that is not your consciousness. And I want to get into the consciousness conversation. But that is not actual presence. And so I think that the value on real human experience, community, communal experience just goes through the roof. And part of that is transparency. That’s authentic.

EL KALIOUBY: So, Natalie, last question. It’s something that I think about a lot and I always ask guests on our show. What does it mean to be human in the age of AI and specifically in the age of AI clones?

MONBIOT: Oh, I’ve been really scratching the surface on this question for the last few months since writing a blog, because I really felt like this is an existential question for me. And I think that in the age of AI twins, when you can offload a lot of stuff to your personal AIs, we need to think about what it actually is to be human. Like, what can’t your AI do? And what your AI can’t do is create meaning. So I think it’s up to humans to be the meaning makers.

Anyway, I was scratching the surface on this question for a bit, and then through this journey of writing, I connected with various people and someone introduced me to this cognitive scientist called John Vey. And I just had this biggest aha moment. If you go on YouTube, there’s a 50 hour course called the meaning crisis — kind of a little heavy, but it’s great stuff.

And basically, what AI can’t do is it’s not embodied. Okay. So we are born to be human. We grow as humans, we’re embodied and embedded in the world. And that is something that an AI can never be. It can learn to see, it can learn to be multimodal — to use AI speak — but it cannot exist, according to cognitive science, as an actual human being. It can never replicate a real human being. At the same time, AI is part of our actual human evolution in a similar way to language. So humans created language, and then language shaped humans and actually gave rise to civilization itself. The introduction of vowels gave rise to democracy in ancient Greece.

So we can’t predict what AI will do, but we invented AI, and if we do it right, we’ll co-evolve with it. And it could, best case scenario, lead to the next era of civilization.

EL KALIOUBY: Love it. Thank you, Natalie, for joining us on the show. What a way to end.

MONBIOT: What a way to end. Tabea.

EL KALIOUBY: There is so much potential when it comes to digital twins. For me personally, it’s been fun experimenting with creating my own AI clones – What do you think of them? Do they look and sound like me?

I’m also excited to put my own AI twin to work! In fact, you may start hearing my AI twin on this podcast. Of course, we will always disclose if we do, but would love to know if you can hear the difference.