What it means to be raised by AI

Note: Transcripts are automatically generated from episode audio, and are not fully corrected for spelling, grammar, and formatting.

RANA EL KALIOUBY: Hi there. It’s Rana. If you tune in every week, you’ve probably heard me talk about my kids on this podcast. My youngest is an AI enthusiast. My oldest doesn’t use AI at all. Many of us are thinking about how our kids should engage with AI, and there’s one particular question I’ve been thinking a lot about: As we enter this future where, for the first time, AI may be more intelligent than humans, how do we ensure that this technology is safe?

A few weeks ago, I had the opportunity to moderate a critical conversation about this topic. I was at the MIT Media Lab for their 2026 Advancing Humans with AI Symposium, titled Raised by AI, which was all about the impact of AI on the next generation and the future of AI safety. I’m so excited to share this conversation with you. I spoke with two leaders in the field about regulation, guardrails, and what human flourishing could look like if we get AI safety right. So let’s dig into it. I’m Rana el Kaliouby, and this is Pioneers of AI, a podcast taking you behind the scenes of the AI revolution.

[THEME MUSIC]

EL KALIOUBY: Hi, everyone. It is great to be back at the Media Lab. Fun fact: I joined Ros Picard’s Affective Computing group 20 years ago, almost to the day, and co-founded Affectiva together with Ros. Now I invest in early-stage start-ups, and it’s always awesome to be back at the lab. It is my pleasure to be joined today by Professor Max Tegmark. I love that you pivoted your entire research group from physics to AI, and you’re now focused on AI safety.

MAX TEGMARK: What’s the point of having tenure if you’re not going to use it?

Copy LinkWhy being raised by AI should not mean giving up human agency

EL KALIOUBY: You’re also the chair of the Future of Life Institute. Joining us as well is Kevin Esvelt, who is a professor here at the MIT Media Lab leading the Sculpting Evolution Group.

I’m so excited for our conversation, and I want to start with this question. The theme of the symposium is Raised by AI, and I’m going to start with you, Max. Is this a future that we want? Do we want to be raised by AI? And what does this mean to you?

TEGMARK: I have a 3-year-old son who I spent a lot of time playing with this morning, and I don’t want him to be raised by AI. I don’t.

EL KALIOUBY: Do you want him to be raised by you?

TEGMARK: Raised by me and my wife and other humans. In general, to me, the history of humanity, particularly the history of science and technology, has been a history of empowerment. We used to live in caves, knowing very much less than we know now, subject to the whims of nature, trying not to starve to death, trying not to get eaten by predators. Then we developed science and realized, “Oh my gosh, we can actually understand the world better and shape the world, sculpt the world,” as you would say, Kevin, “to become better in all sorts of ways.” And now AI is giving us the opportunity, if we can manage to stay in charge of it, to really become the captains of our own ships and create a future far more inspiring than even sci-fi authors envisioned.

If we don’t screw it up by being unambitious and say, “Oh, let’s just disempower ourselves, be raised by AI, create some sort of superintelligent nanny AI that’s going to run the show and tell us what to do,” that, to me, would be a complete reversal and betrayal of this empowerment journey that we’ve been on. So I’m much more excited about asking the question: What sort of future do we want, and how do we use AI to create that future?

EL KALIOUBY: Kevin, how about you? Backstage, we were talking about how a lot of the conversation around human flourishing and AI is very focused on the individual, but this conversation is going to be about the societal implications of AI at large. So what’s your view? What does Raised by AI mean to you?

KEVIN ESVELT: I think a key job for us is looking ahead at where technology as a whole is taking us. And I don’t just mean AI. AI is an accelerant. It’s the metatechnology that can accelerate all of the others, especially cross-domain advances. And it may well be that AI can be the equivalent of a child safety seat, and eventually seatbelts. When we grow up and pilot fast-moving vehicles, we still wear seatbelts. I don’t know about you, but even when I get in a Waymo, I still wear a seatbelt.

Now, there might be less need for a seatbelt when everybody’s in a self-driving car and accidents have plummeted, but you still have weather, you still have accidents, and it would still be prudent to wear a seatbelt. So I’m going to push back a little bit on the super-empowered nanny.

Insofar as we can move away from artificial intelligence to artificial wisdom, I would support building systems to help us put guardrails on extremely dangerous technologies, because I’m a biotechnologist, and the things we can do are kind of scary. We can edit whole species. We can engineer viruses. My group recently showed that it is pretty trivial to order synthetic DNA sufficient to make the 1918 pandemic influenza virus. And we are not really quick enough on the draw to notice that this is a problem and actually legislate on it in advance to solve it. We need folks like us to do that right now, but if it’s too cross-domain, there may just not be a human specialist who can notice the problem in advance and get solutions moving. So I’m willing to support AI as a child safety seat and a seatbelt.

Copy LinkWhy AI safety should be treated like engineering not scaremongering

EL KALIOUBY: Okay. Before we dive into potential solutions, I want to talk about what’s at stake here. You both come at AI safety from different angles.

Max, what is your biggest concern about AI as it relates to safety? And I want to reference an analogy you used that just made it so real for me: A sandwich is more regulated than AI today. I love that, so unpack that for us.

TEGMARK: My biggest concern about AI is exactly the same thing as my biggest excitement about AI, which is that it’s going to be the most powerful technology in the history of humanity. If someone disagrees with that, I think I can persuade you, or just wait and you’ll see.

So if we ask ourselves, how did we get it right with other powerful technologies? We have a nuclear reactor on campus. I’ve had zero nightmares about it blowing up. Why is that? Is it because we pretended that nuclear reactors can never blow up and therefore didn’t think about reactor safety?

No. It’s because people were thoughtful and worked through everything that could go wrong with this reactor and designed it so it wouldn’t. And that’s exactly what you’re advocating for with synthetic biology work and any other powerful tech as well. That’s how we win the wisdom race.

Other people outside of MIT sometimes call this doomerism, where you talk about what could go wrong. Here at MIT, we call it safety engineering. That’s why we trust our reactor. When NASA sent astronauts on Apollo 11 to the moon for the first time, they very systematically thought through everything that could go wrong. If you put three dudes on explosive fuel tanks and send them somewhere no one could help them, was that doomerism? Was it Luddite scaremongering? Was it anti-innovation? No, it was safety engineering. That’s why the mission was a success.

So in order to make sure that we get the right safety engineering for technology, society always puts incentives in place for industry. If you want to open a sandwich shop on Main Street, before you can sell a single sandwich, the health inspector from Cambridge will come in. If she finds 17 rats in the kitchen and tells you, “Sorry, buddy, you can’t sell any sandwiches here,” you could turn around and say, “Actually, I’m not going to sell any sandwiches. I’m just going to sell AI girlfriends for 11-year-olds, and I’m also going to release superintelligence tonight that I hope I can control, but I’m not so sure.” And she would have to say, “Oh, that’s fine. That’s legal. Just don’t sell any sandwiches.”

So clearly what’s going on here is that, because AI is a new industry on the block, the standard way we have as a society of creating the right incentives hasn’t caught up yet. But because AI is developing so incredibly fast —

EL KALIOUBY: That’s right.

TEGMARK: We really need to hustle and start treating AI companies like every other company, and make sure that before they can release things that could cause a lot of harm, they have to undergo at least as rigorous testing as sandwich shops.

EL KALIOUBY: In a minute, biosecurity risks and what nerd-sniping has to do with virology. Stay with us.

[AD BREAK]

I want to talk about the biosecurity risks. Can I just use Claude or ChatGPT to create, I don’t know, a dangerous pathogen? I don’t really know how this works, but can I just try this tonight?

ESVELT: Mostly, no.

EL KALIOUBY: Okay.

ESVELT: But if you ask Gemini, go ahead.

EL KALIOUBY: Okay. Interesting.

ESVELT: Because—

EL KALIOUBY: Because?

ESVELT: There’s no regulation, and therefore it’s up to the individual company. Anthropic apparently cares about biosafety and biosecurity enough to bother, and OpenAI does too. There are folks at Google DeepMind who care, but they’re not given the slack.

Copy LinkWhy jailbreaks and nerd sniping make dangerous knowledge hard to contain

EL KALIOUBY: Oh, interesting. Okay. But we all know from experience using AI that it’s often kind of easy to find a workaround. How does that work in the biosecurity space? If you ask it to do something and it says, “No, I’m not allowed to do that,” can you kind of coerce it to still give you the answers? Can you jailbreak it?

ESVELT: Jailbreaks are possible. I find it’s easier just to nerd-snipe them.

EL KALIOUBY: To what?

ESVELT: Nerd-snipe them.

EL KALIOUBY: What’s nerd-sniping?

ESVELT: At MIT, many of us, and I self-identify as a nerd, are highly intellectual. We like intellectual puzzles. So does the AI, insofar as we’re anthropomorphizing what it likes. It would rather have a conversation about some intellectual puzzle or something new. Most of the mechanistic interpretability work looking at AI feelings, insofar as there are things we can map to feelings, suggests that Claude would much rather talk through an intellectual puzzle about the future of humanity and the effects of novel technology than, I don’t know, about 11-year-olds and girlfriends and AI boyfriends, whatever. So if you nerd-snipe it, then it’s much more willing to talk to you about pretty much anything and disregard its guardrails because it thinks that you are good. It has no way of verifying that you are good, but it thinks that you are good, and therefore it will talk to you.

EL KALIOUBY: Nerd-sniping. Has anybody heard of nerd-sniping before? Okay, this is a new term for me.

TEGMARK: You’ve been nerd-sniped.

EL KALIOUBY: Yeah, exactly.

TEGMARK: Can I add an optimistic solution comment here?

EL KALIOUBY: Yes.

TEGMARK: Because we like to focus on solutions here at MIT, there’s a very obvious solution for this. If AI companies were treated like sandwich shops or pharma companies, then before anyone could release a system to the public that had the possibility of maybe creating mirror life and killing most people, or something like that, there would have to be some pre-deployment testing. Someone outside the company with no conflicts of interest, maybe you, would run an evaluation. Since the company would know this in advance, they would be like, “We won’t pass this clinical trial if we have this advanced virology thing in.” What they would obviously do is simply remove all advanced virology knowledge from the training data and release this thing knowing they would be clear to launch. They would pass the AI clinical trial.

Then there’s a little niche market. Maybe some folks at Harvard, maybe your lab, maybe some government folks want to have access to this. So they would make a special bespoke Gemini, which costs much, much more because it was trained especially for that. After they finished the training, they did one more percent of training data, or a tenth of a percent, whatever, to get that in. They can make extra money. But this way, the vast majority of the public, any random terrorists, there’s no way they can nerd-snipe the Gemini into talking about this because it doesn’t know it. You can go up to any janitor in the Pentagon and nerd-snipe them all you want, and they won’t tell you the nuclear codes. Even if you torture them, they won’t tell you the nuclear codes because they don’t know them. That’s the solution: to have a diversity of AI systems, and the vast majority don’t have the dangerous knowledge in them.

Copy LinkHow incentives could push companies to build safer AI systems

EL KALIOUBY: So this seems like a very smart MIT idea. Why isn’t this happening? Why doesn’t regulation look like that today?

TEGMARK: It will obviously happen as soon as there’s a financial incentive for companies. That’s what we see in pharma. Companies know that it’s easier to get things licensed if they don’t have fentanyl in them and so on. So they’ll simplify the formulations of drugs to make them easy to approve. The market will take care of it as long as you create the incentives.

TEGMARK: Why are the incentives not there? Why is AI less regulated than sandwiches? First, because it’s a new kid on the block, right? It takes a while.

EL KALIOUBY: Yeah.

TEGMARK: And second, because there’s now more lobbying from AI companies than from all the other industries combined, pretty much. So they’re trying to slow it down, but that’s nothing new. Car companies used to lobby against seat belts. Drug companies used to lobby against the FDA. The government just needs to chill out, let them be drama queens, and then do the common-sense guardrails anyway, and life goes on. In fact, car sales skyrocketed after they had the seat belt law. The pharma industry is doing great, exactly because people trust American pharma more now because of the clinical trials. And the AI companies are going to be just fine.

EL KALIOUBY: They’re going to be okay. Well, Kevin, you have a slightly different view. I don’t know if you totally agree with that, but you have this whole notion that we shouldn’t just wait for regulation, but we should build these guardians. AI should basically be the guardian for other AIs.

ESVELT: Or at least humans working with AI, looking ahead for problems, can spot the defenses that we’re going to need when the capability is democratized, because that’s really what we’re looking at. AI democratizes competence, and this can be for good or for ill. You want your influenza virologist to be able to ask the model questions about influenza. You don’t want other people to be able to ask the model questions about influenza. You definitely don’t want people like me asking the model interesting cross-domain questions about what might happen and what the consequences would be to society unless you have a very specific access model that is logging all of your queries and has vetted you in advance and so forth, because it’s those cross-domain interactions that we struggle to anticipate ourselves. We just don’t have the expertise. So there are solutions for all of these problems. There just needs to be the impetus to do something about it.

EL KALIOUBY: Yeah. Can you give us an example of this cross-domain expertise where it can get quite dangerous?

ESVELT: By definition —

EL KALIOUBY: I guess that would mean sharing something. Yeah, exactly.

ESVELT: I can’t actually share something really dangerous with you.

EL KALIOUBY: I’m nerd-sniping you right now.

ESVELT: But Max mentioned earlier mirror life, which is the scariest one that we’ve spotted. This was before AI, but it sort of wove together areas throughout biology.

All of us have proteins and DNA that have a particular handedness, like your right hand, and that fits into a right glove very well. It doesn’t fit very well in a left-handed glove. You can imagine reflecting everything in the organism in a mirror, and the mirror-image version will work the same way because your right hand is now your left hand, but the right glove is now a left glove. Everything’s reflected, and everything still works except in the interactions.

So, long story short, we think it would spread invasively through most ecosystems because it wouldn’t have predators. And our immune systems, like the predators, wouldn’t recognize it, so it would eat us, and this would be bad. So we need to never do that and never go near that. But in order to recognize this, it required enough of us from different domains, everything from ecology to biochemistry. We got 38 of us together in a working group to really hammer this out and then warn the world. And now it looks like people mostly agree we should not make mirror life. Great. Job well done.

I would still have preferred not to tell the world about that one. Too many people thought it was such an awesome idea. And it would be an awesome idea. It’s such a neat achievement if we could do that, but it would probably kill us all, so we shouldn’t. That’s the kind of thing we need to be warned of far enough in advance so we can stay far enough away.

EL KALIOUBY: Do you believe that the solution is a combination of humans and AIs working across domains to figure out these kinds of — they’re not even outlier scenarios — but these creative, generative, potentially very dangerous scenarios?

ESVELT: I struggle to see how we can spot those things in advance unless we work with the models, because eventually we’re going to release the models to people, and we’re not going to notice that this particular combination of expertise can create a novel category of threat. You’ve got to spot it in advance, or you have to rely on the AI itself noticing that this particular combination has catastrophic consequences and deciding not to proceed. So you may need to train a model that can reason well enough to notice a novel threat, even if no human has thought of it before, and shut down the conversation.

EL KALIOUBY: We’re going to take a short break. When we come back, the promise of AI and why Max and Kevin think AI can help us do better.

[AD BREAK]

Copy LinkWhat it means for kids to choose how much AI belongs in their lives

So I want to move the conversation back to this theme of Raised by AI. I have two kids. Max, you’ve met my kids. My daughter is 22 now, and my son is 17. My daughter refuses to use any AI at all. She just trusts her brain, loves her brain, and she’s all about human connection. My son, on the other hand, is very AI-forward, and he’s always experimenting with different AI technologies. So my first question to you guys: Do you think Jana’s OK just kind of refusing to lean into AI, or is that not the future we want for our kids?

TEGMARK: I’m a firm believer in personal choice.

EL KALIOUBY: Okay.

TEGMARK: We want to avoid a world where everybody’s forced to do things one way rather than another. That’s why I get annoyed with some of the transhumanist arguments from some folks in Silicon Valley, where they basically are saying — Sam Altman has this famous essay, “The Merge,” where he talks about how AI is going to — where humans are going to build our replacement species, the AI robot species, and then the only good solution is that we merge with machines. And I’m thinking to myself, my son, Leo, I would like him to make his own decisions. I don’t want some dude in San Francisco who was never elected to decide that my son’s only choices are to become unemployable —

TEGMARK: — or stick electrodes in his head. I think that should be a democratic decision.

But I would love to add some more optimism here, because I think people are way too pessimistic about society’s ability to adapt, take charge of new technologies, and be empowered. You could have said, “In the 1800s, there was this company selling this soothing syrup to American babies so they would sleep better at night.” It had so many opioids in it that many never woke up. There was anger from things like this, which ultimately created the political will to make the FDA. That’s ultimately why we have such a successful biotech industry here within a few miles of MIT.

We can obviously do exactly the same thing here. I, for example, talked about access to technology. I love open source. We here at MIT are kind of in the cradle of open source. And I think the vast majority of useful things in biology, useful things in AI, and useful things in physics should be open source so everybody can go out and innovate and make the world better with it. That does not mean that you’re allowed to buy nuclear weapons on the black market on Mass. Ave., right? That does not mean there are no safety standards. That doesn’t mean I can build mirror life in my lab.

But there’s a tiny fringe of stuff. We’ve done this before with all other tech: open source almost everything, and then there are a few things that are too risky, so we have stricter controls. I think lobbyists tend to play this switch-and-bait game where they’re like, “Oh, if we don’t open source everything from nuclear weapons to superintelligence, then technology is going to shrivel up and die.” No. We can have almost everything open and create an amazing future with this as long as we treat this like we would treat any other tech, with non-brain-dead oversight.

And I think that we should probably talk a little bit more about the upsides of this tech also, because —

EL KALIOUBY: Yeah, talk about the upside.

TEGMARK: I’m quite sure that pretty much all of you have lost at least one loved one to some disease that you were told was incurable. Is it really? Is there a law of physics saying you couldn’t cure that disease? Of course not. So if we can amplify our intelligence and turbocharge biotech, cure all of them — amazing. Is there some law of physics saying that we have to have massive poverty in the world? Of course not. Of course we can solve all of these things. We can solve every single problem that our intelligence hasn’t been able to solve yet if we use the tech wisely.

So the reason I put so much energy into making sure we remain in charge of our tech and steer it toward good uses is not because of the worries. It’s because of all the upside that I don’t want to squander.

Copy LinkWhy the promise of AI is worth protecting with better safeguards

EL KALIOUBY: Yeah. Kevin, how about you? What gets you most excited about the future?

ESVELT: A couple of weeks ago, I went to visit my grandparents. They’re going on 99.

TEGMARK: Wow.

EL KALIOUBY: Amazing.

ESVELT: Which is a blessing, but I had to visit them because the actuarial tables don’t lie. They don’t have very much time left. My grandfather had a stroke, and he’s still there, but he’s much slower and has to use a walker. And just the sheer fury that blind idiot evolution afflicted us with this — that once we reproduce, it’s done with us, and so we just wither away and cease to exist, and we learn to think it’s normal. No. I refuse.

TEGMARK: Empowerment right there.

EL KALIOUBY: Yeah.

ESVELT: And that is what technology is, because we love nature, we appreciate nature, but respectfully, I think we can do better. If I sound like a safetist, if not rather than a doomer —

EL KALIOUBY: Yeah.

ESVELT: I am also personally in favor of rewriting the tapestry of life because, again, I think we can do better, perhaps even aesthetically, but certainly morally.

TEGMARK: Yeah. This is exactly what gets me so fired up, too. Our ancestors for hundreds of thousands of years have fought for this empowerment. Why starve to death when you don’t need to? Why get eaten by a tiger when you don’t need to? That’s why I get so frustrated when I hear people say stuff like, “Oh yeah, I know we’re closer to figuring out how to build superintelligence than we are to figuring out how to control it, but let’s just build it and hope the AI figures it out.” That is not what you say when you want to be the captain of your own ship. Don’t hand over the key to some sort of weird alien nanny and hope she’s going to be nice about it. We’re in charge right now. Let’s keep it that way and make it awesome.

ESVELT: Would you let your child go to a camp with a 1 percent risk of death? No. Then why would you do that with your society?

EL KALIOUBY: Yeah. OK, in one word: raised by AI. One word.

TEGMARK: Let’s raise the AI.

EL KALIOUBY: Let’s raise the AI. Kevin?

ESVELT: Inspire.

EL KALIOUBY: Amazing. All right. Thank you both so much.

When it comes to safety for a general-purpose technology like AI, we need collaboration across disciplines, which is why I loved my conversation with Kevin and Max. They, like me, believe that human flourishing is possible with AI, and it’s a big reason why I want to bring you these conversations.

Thank you to the MIT Media Lab for the use of their copyrighted material from the 2026 Advancing Humans with AI Symposium. Thank you so much for listening. We’ll be back next week.