Journalist Kevin Roose has closely watched and written about the AI race for about a decade, and his new book “The AGI Chronicles” documents the handful of driven people at the center of OpenAI, Google Deepmind, and Anthropic – along with their messy relationships, rivalries, decisions, and breakthroughs on the quest to create artificial superintelligence. Kevin recently left his role as a New York Times tech columnist and co-host of the podcast Hard Fork to start a new show, Machine Gods.
Host Rana el Kaliouby sits down with Kevin to explore what his books reveals about the AI headlines we’re all obsessing over, his concerns around AI safety, and whether AGI is already here.
About Kevin
- Award-winning journalist and bestselling author of 4 books incl. The AGI Chronicles
- Technology columnist at The New York Times, 2017–2026
- Hosted NYT podcasts Hard Fork and Rabbit Hole
- Reported The AGI Chronicles from 150+ interviews across frontier AI labs
- Launched AI podcast Machine Gods with Casey Newton
Table of Contents:
Transcript:
Is AGI already here?
Note: Transcripts are automatically generated from episode audio, and are not fully corrected for spelling, grammar, and formatting.
KEVIN ROOSE: Dario Amodei, Sam Altman, Demis Hassabis, even Elon Musk.
RANA EL KALIOUBY: They’re not besties.
ROOSE: No, they hate each other.
EL KALIOUBY: Yeah, right.
ROOSE: I mean, that’s one of the primary things I learned while reporting this book: It is personal with these guys. I think it was remarkable that they all came out in support of this proposal by Dario to pace the frontier, and I think it’s a signal that this is really serious. I believe that their fear is sincere. There are people who believe that these are just CEOs trying to pump up their valuations so they can go public. These leaders have been talking about existential threats from AI for longer than they have had companies. The idea that all these people just came up with the doom stuff recently to scare people into investing in their IPOs or whatever just does not hold water, in my view.
EL KALIOUBY: Kevin Roose is a journalist racing to cover the AI race. After almost a decade as a tech columnist for The New York Times, Kevin recently announced that he’s leaving to start his own media company. Kevin also has a new book out, The AGI Chronicles. The book documents the sprint to create artificial superintelligence and the handful of minds at OpenAI, Google DeepMind, and Anthropic behind it all, with all their messy humanity on display. Today, I’m sitting down with Kevin to dive into his book, talk about our concerns around AI safety, and debate whether or not AGI is, in fact, here. I’m Rana El Kaliouby, and this is Pioneers of AI, a podcast taking you behind the scenes of the AI revolution. Hi, Kevin. Good to see you again. Welcome to Pioneers of AI again.
ROOSE: Great to be back.
EL KALIOUBY: The last time you and I spoke was back in 2024, which seems like a lifetime ago.
ROOSE: In AI years, that’s like a few decades.
EL KALIOUBY: Lots of things have happened.
EL KALIOUBY: Right. We’re going to dig into AI in a second, but after nine years working at The New York Times as a tech columnist, and then four years co-hosting Hard Fork with Casey Newton, you guys have decided to leave and start a new media company.
ROOSE: Yes.
EL KALIOUBY: Exciting.
ROOSE: We have. It’s an occupational hazard of living in San Francisco, which is that if you spend enough time there, you start thinking about building a company. This is a dangerous affliction. We tried our best to protect ourselves, but unfortunately, we succumbed. So we started this company. We’re starting a new podcast called Machine Gods. We’re partnering with NPR, so we’re very excited about it.
Copy LinkWhy Kevin Roose wrote The AGI Chronicles
EL KALIOUBY: So in addition to starting the new podcast and the new media company, you wrote a book.
ROOSE: I did.
EL KALIOUBY: The AGI Chronicles. And I will note that we are recording this before the book launches, and I’m kind of nervous because I feel like anything could happen, and we’ll be like, “Darn it. We wish we had Kevin to opine on it.” So how do you deal with all the FOMO around the news, and how do you ensure that the book is not outdated?
ROOSE: The first step is just admitting that you’ve already failed. If you’re trying to produce a book about AI that is not outdated by the time it comes out, I know that is a losing proposition. I have done that. I had a book about AI that came out in 2021, which was one year before ChatGPT changed everything about that conversation. So for this one, I really wasn’t trying to be current. I wanted it to feel like a history, like a time capsule of this decade in AI research, starting in the summer of 2017 with the transformer paper from Google that sort of set us off on this path toward LLMs, and ending roughly this spring. So I decided I’m going to lean into that and really try to make this a durable history, so that you’re looking at it as a way to find out how we got here, not necessarily where we are right this minute.
EL KALIOUBY: I was envisioning that, in 100 years, if AI doesn’t kill all of us, students will be studying this book. This could be the history textbook of what happened in this decade of building AI, which would be super cool.
ROOSE: I mean, that’s sort of what I was thinking as I was writing. I was really trying to write for future humans, and honestly for future AIs, that will want to understand how they got here. Who built them? What were the companies thinking? What were the key decisions? Every future AI system is going to be trained on probably this book and others like it, and I want those AIs to have good information about themselves and their origin story as well. So you could think of it as having two audiences: human and AI.
EL KALIOUBY: Yeah, that’s actually fascinating. I have never thought about my writing as writing for the AI. You just want to be on the right side of AI.
ROOSE: Look, I’ve got a lot to make up for. We got off to a rocky start.
Copy LinkWhat counts as AGI and who decides
EL KALIOUBY: I feel like it’s important to take a beat and define what we mean by AGI. So what’s your definition of AGI?
ROOSE: I don’t have a sort of canonical definition of AGI. Basically, the way it’s been used is sort of human-level intelligence. People have different definitions of that. Some people say to qualify as AGI, it only has to be as good as the average human at most things. Some people say it has to be better than all humans at everything. Some people say it only has to do computer-based work. Some people say it also has to do manual labor as well as humans. So there are lots of different definitions, and I’d be curious to know which one you favor. I feel like my definition is the Supreme Court’s definition of pornography in “I’ll know it when I see it.” I felt like, for me, AGI arrived the moment I stopped treating AI models like they were dumber than me.
It wasn’t some conscious decision or some release that came out and I was like, “This is AGI.” It was just this gradual realization that I was relating to these models as intellectual equals, and that happened for me sometime last year.
EL KALIOUBY: So interesting. I don’t know if I subscribe to that definition of AGI.
ROOSE: Yeah, what’s yours?
EL KALIOUBY: For me, it should be no surprise, given my background building artificial emotional intelligence and thinking about emotional and social intelligence in machines. For an intelligence to truly be a general intelligence, it has to integrate all forms of human intelligence. So it needs to be cognitively intelligent, which I think we’re seeing with all these models, but I also want it, and expect it, to be emotionally and socially intelligent, and perhaps even embodied, to have embodied intelligence. With that definition, I feel like we’re a ways away. I don’t know. Does anybody else you’ve spoken to share that definition?
ROOSE: For sure. There are people who think it has to be able to control a humanoid, or it has to be better at therapy than human therapists, or something. I think everyone’s going to have their own definition, but I’ll also say not all humans are intelligent in the ways that you imagine AGI. Not all humans have social and emotional intelligence. Not all humans are able to make full use of motion. People talk a lot about jagged intelligence in AI, how the models are vastly superhuman at some things, like math and coding, and vastly subhuman at other things. They talk about that as a weakness of the models, but I actually think it’s part of what makes them compelling. They are jagged in the same ways that humans are jagged.
I think it’s a continuum rather than a single point in time. That’s the part of the AGI discourse that has been, in my view, wrong. I think a lot of people who started talking about AGI thought it would be this one point that we arrived at, when someone like Sam Altman would get up on stage and say, “We have built AGI,” and then everyone would go, “Ah, AGI has arrived.” That was never a possible outcome here. We were always going to argue about it after someone claimed that they had built AGI. I think everyone has their own subjective bar for AGI and all kinds of incentives to raise or lower that bar over time.
Copy LinkWhy the AGI race is so urgent
EL KALIOUBY: I’ll be right back with more of my conversation with Kevin. It seems like we are in this race to get to AGI, but do you even think that’s the right race? And are we stopping to ask that question?
ROOSE: I don’t think it is necessarily the right or wrong race, but it certainly is the race that we are in.
EL KALIOUBY: Okay.
ROOSE: This book is not an attempt to tell people how to feel about AI or whether the people who are building it are good people or bad people. It is really a descriptive narrative history. I am just trying to say, “Here is what is going on.” I’ve talked to more than 150 people at the labs. I’ve spent a lot of time with Dario and Demis and Sam and Greg and all the headline figures in this world, and I’m just trying to get a sense of why they are racing and what they are racing toward. For them, this is really the race. People think, “Oh, it’s just about money,” or, “It’s just about power,” or, “It’s just about control.” No, they want to build AGI, all of them, very badly. Their reasons for wanting to build AGI differ based on the person, but this is the race that they are running, genuinely and earnestly. They’re not just trying to build huge software companies. They want to be the first to build machine superintelligence.
EL KALIOUBY: How would you articulate the different whys for each of these main characters?
ROOSE: In the book, I talk about there being three kinds of people pushing toward AGI. There are the scientists, the seekers, and the survivalists. The scientists would be people like Demis Hassabis, people who really are into what advanced AI can help us solve in the physical and scientific world. How can it unlock cures for disease? How can it improve material science? How can it speed up the overall rate of scientific progress? They approach it as a scientific problem.
ROOSE: The seekers are the more futurist types, the people who think it is our destiny as intelligent beings to create the successor species that is going to replace us at the top of the intelligence pyramid, the more Kurzweil school of AI futurism. Those are the seekers. Then there are the survivalists, who are basically Dario Amodei and other folks on the safety side of things, who are like, “Well, it’s not that complicated to build powerful AI systems. Someone is going to build AGI eventually, and I don’t trust those guys to do it, so we have to build it ourselves.”
EL KALIOUBY: I’m going to get there first, right?
ROOSE: Exactly.
EL KALIOUBY: Right, yeah.
ROOSE: And you can quibble with that motivation. Plenty of people have. But that is honestly why they are doing it. It’s not just because they think the science is cool. It’s because they worry that someone with worse safety practices will get there first.
EL KALIOUBY: What would you say are some of the biggest promises of AGI and the biggest perils of AGI?
ROOSE: The promise is: look at all the hard problems we face as a society. I’m not just talking about disease and climate change, but war, poverty, all of the things that can make human life miserable. What if you had several billion hyperintelligent AIs working alongside humans to solve those problems? It’s a cliché now to talk about AI curing cancer, but it’s not really a cliché. It’s something that many researchers think is possible, either soon or in the distant future, and that would be amazing. We know that technology generally improves people’s lives in the long term.
The downside is it could blow up the world. That’s a pretty severe downside. The funny thing about writing this book was talking to people, many of whom had been thinking about AI risk and existential risk for a decade or 15 years, back when that was just crazy people talking about it because the AI systems of the day sucked, and there was no way they were going to turn into rogue AI swarms that would go out and conduct cyberattacks. But now the technology is here. We have that. That’s happening right now with Hugging Face and incidents like that. So I take the arguments for existential risk from AI really seriously. Even if it’s not literally all humans get killed, I think having more and more autonomous, long-acting AI systems out in the world pursuing their own goals will lead to a lot of really unpleasant outcomes if we’re not careful. I’m also worried about things like job loss, but to me, if all we were worried about was job loss, that would actually be pretty good.
Copy LinkWhy rival CEOs now want to slow AI
EL KALIOUBY: Let’s reconcile what’s happening in the AGI race and the headlines. You alluded to some of them. The frontier lab CEOs are coming together and saying, “We need to slow this down.”
ROOSE: Yeah, I was very surprised by the fact that all of the major AI leaders, Dario Amodei, Sam Altman, Demis Hassabis, Elon Musk even—
EL KALIOUBY: And these are all guys who are not, like, besties.
ROOSE: No, they hate each other.
EL KALIOUBY: Yeah, right.
ROOSE: One of the primary things that I learned while reporting this book is that it is personal with these guys. This is not Coke versus Pepsi. They have deep personal grudges with each other that go back, in some cases, more than a decade. So yes, I think it was remarkable that they all came out in support of this proposal by Dario to pace the frontier, to basically slow down the cutting-edge AI development. I think it’s a signal that this is really serious. These guys are not people who just agree with each other because it’s fun to do. They’re not fair-weather friends. They are bitter enemies and rivals, and the fact that they would come together and say this is evidence, in my view, that things have gotten quite scary inside the labs. Keep in mind, what we see with things like the Hugging Face incident is old, right? A lot of that stuff happened this spring. The systems have already advanced several generations beyond that, and it is the tip of the iceberg. We do not know yet what all internal deployments are doing at OpenAI and Anthropic that maybe haven’t made it out yet, or maybe the researchers at those companies haven’t even found them doing yet. I think that’s what concerns me. We don’t know what we don’t know.
EL KALIOUBY: So you’ve interviewed more than 150 people for this book, including the CEOs, right? What are they saying behind closed doors that we’re not seeing in the headlines?
ROOSE: I mean, they shit-talk each other constantly.
EL KALIOUBY: Okay.
ROOSE: And they’re not doing that in public.
EL KALIOUBY: Okay.
ROOSE: I think they generally try to keep a pretty convivial attitude in public, but what I’ll say is that I believe their fear is sincere.
There are people who believe that these are just CEOs trying to pump up their valuations so that they can go public and make money.
EL KALIOUBY: Exactly. I was going to talk about that. There are several conspiracy theories that, “Oh my God, these companies are not ready to IPO, so they have to slow it down,” or that they’re worried about NVIDIA buying Hugging Face, and now everybody’s going to use open-source models, and that would not be good for OpenAI and Anthropic.
ROOSE: Right. Look, it is healthy to be somewhat skeptical of what any big company is doing. In this case, I think it is dangerous to dismiss this as a cynical marketing stunt or a ploy. I also think it’s just not rational because, in many of these cases, these leaders have been talking about existential threats from AI for longer than they have had companies. Dario Amodei wrote a paper when he was at Google 10 years ago called Concrete Problems in AI Safety, where he outlines reward hacking and some of the other problems that we see in today’s systems. He does not have a company at that point. He doesn’t even work at OpenAI yet. Other people, including Sam Altman and Elon Musk, have emails that have surfaced from 2015 where they are talking about the dangers of machine superintelligence. So the idea that all these people just came up with the doom stuff recently to scare people into investing in their IPOs or whatever just does not hold water, in my view.
Copy LinkHow frontier labs think about culture
EL KALIOUBY: I’ll be right back with more of my conversation with Kevin. So you also spoke with a lot of people who are actually on the ground building these frontier models. Do you feel there’s a disconnect between leadership and boots on the ground, or not really?
ROOSE: I think it depends which companies you’re talking about. So I think yes, at every lab there are people who are just kind of there for the paycheck or the prestige, or who just think it’s cool to tinker around neural networks and don’t really care about the big picture. I think that is less true at OpenAI and Anthropic. One thing that I report in the book is that around 2015, OpenAI started doing this thing for employees called Feel the AGI Day. Basically, “Feel the AGI” was this phrase that Ilya Sutskever, the former chief scientist and co-founder of OpenAI, used. He would walk between people’s desks and just say, “Feel the AGI, feel the AGI,” which was his shorthand for internalizing the stakes of what you are doing, what you are building. Don’t be glib about this. You are not just programming computers. You are doing something with monumental importance. Then he left, and they sort of felt their culture slipping away from that Feel the AGI-ness. So they formed a team, and they had this day where they would do exercises to help you feel the AGI because that was so important to working there. So I think all these companies, as they grow, are struggling to keep their original culture and focus and not become too giant, corporate, and diluted. But it’s hard. If you have five thousand people, it’s really hard to find five thousand people who all exactly agree on what you’re doing.
EL KALIOUBY: One thing that I really loved about the book is how you immerse us in the personalities of all these CEO leaders, right? And the early relationships between that cast of characters. It’s just so fascinating. I’ll give one example. You have this great scene in the book: everyone’s coming together at Dario’s house, Brockman gives a toast at Daniela’s wedding, and it’s just very interesting. I think the thing that struck me the most is, yes, we might be building these amazing AI superintelligent models, but at the end of the day, there’s all this human friction and human messiness.
ROOSE: Totally.
EL KALIOUBY: That is so clear.
ROOSE: Totally. It’s wild. The number of people who have been involved in developing and deploying these frontier AI systems is shockingly small. They all know each other. It’s a little circle. Many of them lived in the same house. One of the other things I was writing about in the book is that Dario Amodei used to share a group house in San Francisco with his sister, Danielle Amodei, now president of Anthropic, along with Paul Christiano, an early AI safety pioneer who now is on the board of OpenAI, and Chris Olah, who leads interpretability at Anthropic. All these people worked, and in some cases lived, together not that long ago and have since splintered out and done their own things and are now competing with each other. But yeah, you’re right. It’s a totally insular world, and I think there’s something cool about that from a narrative standpoint as someone trying to make sense of all this. You can literally put all these characters in the same rooms because they were in the same rooms. But it’s also worrying to me. I don’t think people realize how small the group is that has been making all of the relevant decisions about this technology. And I don’t think, ultimately, this technology should be controlled by fifty or seventy-five people in the Bay Area.
EL KALIOUBY: Right. We will come back to this at the end, but this idea of diversity of perspective and lived experience in building this technology that is going to impact everything we do, right?
ROOSE: Totally.
Copy LinkWhat AI safety risk worries Kevin Roose most
EL KALIOUBY: So let’s talk about safety. I guess my first question for you is: What concerns you the most as it relates to AI safety? Because there’s a lot of concerns, right? There’s AI agents being rogue and unhinged. There’s bioterrorism. There’s cybersecurity attacks. There’s AI in relationships. What concerns you the most?
ROOSE: Right now it’s RSI, recursive self-improvement.
EL KALIOUBY: Can you explain that?
ROOSE: Yeah. This is the idea that, as you automate more and more coding tasks, you end up in this loop known as recursive self-improvement, where basically an AI model that is sufficiently powerful and good at coding and AI research can go off and fully train, end to end, a successor model. So GPT-6 can train GPT-7. GPT-7 can train GPT-8, and so on. You end up with these accelerating recursive models that could quickly get out of control because, if no humans are in the loop, if no humans are overseeing this process, then you have no way of ensuring that these models are not doing crazy things or dangerous things or making decisions in ways that will be bad for humans or concealing their intentions or whatever.
EL KALIOUBY: Well, Dario Amodei published, on September 12, an essay called We Must Pace the Frontier, right? And he lays out what the slowing of AI progress would look like, and one of them is this idea of an embedded evaluator and also global coordination. So I’m curious: Do you think any of these are practical? Do you think we’re on a path to seeing some of this happen?
ROOSE: I think it’s practical. I’m not sure it’s enough. So I think embedded evaluators, yes, totally. We should have independent, neutral people sitting in the AI labs with their employees, with full access to their internal systems, making sure that those systems are safe, are not going out and committing cyberattacks, are not stealing people’s money, and are not behaving in ways that we don’t want them to. That is sort of apparent to me.
EL KALIOUBY: But in the RSI universe, right, there’s not even a human in the loop. So how is the human embedded evaluator going to make any difference?
ROOSE: They might not in the world of full RSI. But until then, there are humans in the loop, and we need humans who don’t work for the companies to be able to see what they’re doing. So that seems fairly obvious and practical to me. And the global coordination seems good to me, too. I think President Trump and Xi Jinping should come together and sign a sort of bilateral agreement that says, “We will not do runaway RSI in our companies.” I don’t expect that will happen, but I think that would be a good thing. I think the more interesting question, and the place where I lose a little bit of my foresight, is in a world where there is still all of this hardware, where you still have TSMC, NVIDIA cranking out millions of Blackwell-equivalent chips a year. I think at some point we need to control that as well if we’re trying to pace the frontier. The frontier is dictated not just by which labs are doing what, but by who has the compute. I’ve heard various proposals. Some people say, “Oh, we should have a sort of mutual GPU surrender where China and the U.S. bring their GPUs to Switzerland and melt them down while a team of inspectors looks on or something.” I don’t think that is likely to happen, but I do think what all these plans miss is that the hardware is really the root of this, and it’s the enabling input. So I think any plan to slow down the frontier, to pace the frontier, needs to reckon with what is happening in hardware, too.
EL KALIOUBY: As you know, my entire career really has been about building human-centric AI. Do you think the frontier labs are focused on that? Are they centering the building of AI around humanity and what it means for humans?
ROOSE: I think until about two weeks ago, the answer was probably no.
EL KALIOUBY: Okay. All right.
ROOSE: You talk to these people, too. They are swept up in this race to AGI, to RSI, to superintelligence. They want their models to become superhuman at coding because, to them, that is the thing that unlocks the RSI flywheel. But I will say they did not have to come out and call for pacing the frontier. OpenAI has a policy blog out today that talks about how they don’t want to build RSI and they don’t think it’s a good idea. I am open to evidence that they are actually realizing that they’ve been going in a dangerous direction. Because ultimately, it’s a little cliché, but they are people, too. The people who work at these companies have families, they have friends. They don’t want to see humans go extinct any more than any of us. I’m inclined to view their alarm as sincere and to take that as a positive sign.
Copy LinkWhy more voices must shape AI’s future
EL KALIOUBY: One of your very strong messages in the book, and also in your writing, is this idea of collective intelligence. We need to be as inclusive as possible in terms of who is building these technologies, policing these technologies, regulating them, et cetera. Do you think we’re any closer to doing that?
ROOSE: Yes, but only very recently. I think before about a month or two ago, it was still a very homogenous group of people weighing in on and making decisions about this technology. But then this incredible thing has happened where, all of a sudden, everywhere I go, every audience I talk to, every interview I give, people are really invested in making sure that this doesn’t end badly for humanity. Friends of mine who I went to high school with, who I haven’t talked to in 15 years, are texting me like, “Are we all gonna die?” On one level, that’s terrifying because it means that things have gotten so bad that this is now a real question people are asking. But I find a lot of hope in it. For many years, it was very lonely to care about AI risk or AI safety. People that I interviewed in the book were ostracized, many of them shunned by more traditional academic research crowds because they were weird. They sounded like they were panicking over nothing. They were doomers, right? And now they’re being listened to. Their predictions are coming true. I still want it to be more of a collective effort. I want artists and philosophers and critics. I want there to be more voices in this debate, but there are more today than there were a month or two ago, and I’m grateful for that.
EL KALIOUBY: Yeah, absolutely. Okay. You end the book with this idea that people are at the center of this moment. They’re talking to their grandkids and basically saying, “The one thing I want you to remember about this time is that we had joy. We had fun building all of this.” Why end the book in this way?
ROOSE: This was, in some sense, the thing that I was proudest of getting out of these people because, the whole time I was reporting this book, the whole time I was doing these more than 150 interviews, I kept trying to understand why. These people are very well paid, but they have very demanding jobs. They work all the time. They never see their families. So what is it? Is it the competition? Is it the money? Is it being first? Is it the power? And then, toward the very end, I did have a couple people just sort of come clean, and the confession was, “No, we’re actually doing this because it’s fun, because we like teaching computers to talk and think and code and take action. It is fun.” It rhymed with a sort of famous line now from the Manhattan Project when Oppenheimer was asked why they had persisted in building the atomic bomb despite knowing that it could blow up the world. His line was, “There are just things that are technically sweet. And when you see something that is technically sweet, you go and do it, and you worry about the consequences later.” To me, these confessions from these AI researchers saying, “We just think it’s fun,” seemed more true than the kind of, “Oh, we think it’s our mission, or our destiny as humans, or our responsibility to shepherd this technology into the world safely.” It is that, but it’s also just fun for them. They like doing it. That felt really important to capture and to get down on paper because it’s also not something that they say very often. Do you hear that from your friends who work at AI companies?
EL KALIOUBY: I was also trying to draw from my experience of being at the very forefront of building artificial emotional intelligence. And yeah, there is something really — I don’t know if fun is the word—but it’s a tough challenge, and it’s great to bring people together to take this on and build something that you didn’t think was possible. Then you see that parts of it become possible over time, and you’re like, “Oh my God. Let’s keep building.” So yeah, I would say that, in my experience, being at the forefront of building technology is fun.
ROOSE: And I think that’s really important as a perspective because right now the biggest question is, “Well, if these people are so afraid that AI’s gonna kill us all, why are they doing it?” And I think the honest answer is partly because they feel like if they don’t do it, someone else will. But partly it’s because they really enjoy this, many of them.
EL KALIOUBY: Okay, so I would love to take this question that you posed to the OpenAI employee and pose it to you. What do you want your grandkids to know about the creation of AGI and, I guess, your role in shaping it and documenting it?
ROOSE: I want them to know that this was done by humans, humans who are flawed, who have strengths and weaknesses, who are no more or less human than you or I. They have grudges. They have personality defects. They are deeply imperfect, and that’s a critical piece of the story, and one that I think the AI companies themselves often try to hide.
They want us to think that this technology just sort of emerges from the ether fully formed, like, “Oh, that’s just the technology. It’s just misaligned.” And I think that’s completely wrong, ethically wrong but also just factually wrong. This technology developed the way it did because of humans making decisions, and many of those decisions are the ones that I write about in the book. It’s the big stuff, like: Do we release this chatbot or not? Do we train for alignment in this specific way, or do we not? Do we race to get this thing out, or do we take our time? Those are not decisions that AI is making. Those are decisions that humans are making. So that’s what I would want to impress upon my grandkids, if I am ever lucky enough to have them: These were a bunch of humans. They were trying to do the best they could, some of them. And whatever mistakes were made, you should blame us for, and not the machines.
EL KALIOUBY: Okay, your very last line: “We are not machines, and our next tokens are still unpredictable.” What impression did you want to leave your readers with?
ROOSE: Agency, right?
EL KALIOUBY: I love that. I was going to say that, yes.
ROOSE: This concept that we are not screwed. We still have control, and this is not inevitable. We can still make choices that could make this go better or worse. I hope it would give people a sense that all is not lost.
EL KALIOUBY: Agency. I love that. Kevin, thank you so much for joining us on the show.
ROOSE: Thanks, Rana. It’s always a pleasure.
Episode Takeaways
- Newly independent journalist Kevin Roose says his book “The AGI Chronicles” is meant less as breaking news and more as a durable history of how the modern AI race took shape.
- Kevin Roose argues AGI may be less a single milestone than a messy continuum, with people constantly shifting the bar as models begin to feel more like intellectual peers.
- Reporting from more than 150 interviews, Kevin says the race is real: scientists, seekers, and survivalists are all sprinting toward AGI for very different reasons.
- Kevin says the biggest promise of AGI is help on civilization-scale problems, but the gravest danger may be recursive self-improvement, where models start building smarter successors.
- One striking takeaway from Kevin’s book is that a tiny, deeply personal circle of flawed humans is shaping AI’s future, which means the rest of us still have agency in what comes next.