How today’s AI giants mirror global empires of the past

Note: Transcripts are automatically generated from episode audio, and are not fully corrected for spelling, grammar, and formatting.

KAREN HAO: When the term artificial intelligence was coined in the first place, it was coined largely as a marketing ploy. Because this Dartmouth assistant Professor John McCarthy, he said many years later.

JOHN MCCARTHY: I invented the term artificial intelligence. I invented it because we had to do something when we were trying to get money for a summer study.

HAO: His mentor actually heavily pushed back against that term saying, no one’s gonna know what that means. It’s gonna oversell this technology, it’s gonna create all this confusion. And of course it did, but the term also ended up just sticking. And now we have the term AGI, replicating all the same problems and confusions.

RANA EL KALIOUBY: Karen Hao wants to cut through some of that confusion. Her new book, Empire of AI, chronicles the rise of generative AI through the lens of the goliath companies behind it. She likens their rise to imperial empires of the past. After years of reporting on OpenAI, Karen lays out the history of the company and offers her own critique about where AI is headed.

Her book is a must-read for AI skeptics and enthusiasts alike. And I’m so excited to share our conversation.

I’m Rana el Kaliouby and this is Pioneers of AI, a podcast taking you behind the scenes of the AI revolution.

[THEME MUSIC]

EL KALIOUBY: Hi Karen. Welcome to Pioneers of AI.

HAO: Thank you so much for having me, Rona.

EL KALIOUBY: So we share the MIT connection and back in the day when you were at MIT Tech Review, you covered my company Affectiva a few times. So it’s great to reconnect after all these years.

HAO: Yeah, it’s really great to speak again.

Copy LinkWhy belief became a defining force in OpenAI's story

EL KALIOUBY: Okay. We are gonna talk about your book, Empire of AI. Congratulations. And of course the book became an instant New York Times bestseller. So, congratulations. How are you feeling?

HAO: Really grateful for the reception that it’s been getting. It’s what I dreamed of, so it’s amazing to see it happening.

EL KALIOUBY: So I have to ask you this, is this all gonna become like a Netflix TV series or something? I can’t wait for that.

HAO: That would be my dream. Any Netflix producer listening to this, call me.

EL KALIOUBY: Exactly. All right. Let’s see if we can make it happen. Joking aside, one of the things I always do when I pick a book is I go to the acknowledgement section, and I read that first, and I was actually really struck by yours because you start with this idea of belief, this leap of faith that we all take when we have deep conviction in an idea or a product or even ourselves, right? So I wanna ask you this. Why is the concept of belief so central to both your personal story and the theme of the book?

HAO: Yeah, I was really struck by the fact that throughout my career I have had just amazing people around me that have really believed in me, and that has been such a core aspect of being able to do the things that I have wanted to do. But it is also such a central part of OpenAI’s story in that it can also become a toxic thing. I opened the book with this quote that Sam Altman found when he was a young entrepreneur.

He ended up writing about it in a blog post in 2013, and it goes like this. Successful people build companies. More successful people build countries. The most successful people build religions. And then he’s reflecting on that saying that the most successful founders don’t seek to build companies.

They seek to build religions, and ultimately building a company is the best way to do so. It felt like belief is such a core theme that runs through OpenAI’s history as well. Just belief in AGI, belief in oneself, belief in a religion — it’s such a fascinating core aspect of the OpenAI story, but it’s a double-edged sword sometimes.

Copy LinkHow AI giants mirror the behavior of historical empires

EL KALIOUBY: Yeah, absolutely. And actually one of your primary arguments in the book is that these big AI companies like OpenAI are modern day empires, and you compare them to the European imperial powers that colonized so many countries. Walk us through this argument and why do you make this analogy?

HAO: Yeah, so there are four parallels that I draw between empires of AI and empires of old. The first one is that empires lay claim to resources that are not their own, but they reinterpret the rules to suggest that those resources were always their own. And that refers to the way that companies scrape the data on the internet.

And they say that it’s totally fair game because it’s in the public domain. But of course, the people who put their data on the internet, they didn’t give informed consent for their data to be scraped and used to train models that ultimately might constrain their future economic opportunity. It also refers to the fact that these companies take the intellectual property of artists, writers, and creators without credit or compensation and say that it’s fair use under copyright law. Empires also exploit a lot of labor and that refers not just to the labor exploitation that happens in the AI development process, which I document extensively in my book, but also the fact that the production of this technology is inherently a labor automating one in that OpenAI’s definition of AGI is.

Highly autonomous systems that outperform humans in most economically valuable work. So they explicitly say that they’re trying to build systems that are automating away the tasks that people usually get paid for. So that in and of itself then erodes workers’ ability to bargain, to negotiate, and that’s going to lead to labor exploitation as well.

The third feature of empire is that they monopolize knowledge production. So in the last 10 years, what we’ve seen is the AI industry has become so resource rich that they’re able to offer compensation packages that easily cross over a million dollars. And so the top AI researchers in the world have moved from working for academia or independent institutions to working for these companies. And the effect would be exactly the effect that you would imagine if most climate scientists in the world were bankrolled by oil and gas companies, you would not get a clear picture of the climate crisis. And that is essentially what’s happened with AI research.

Companies are actively censoring the research that is inconvenient to them. So we are not getting a full picture of the true limitations of this technology as it’s being deployed into the world. And then the fourth and final feature of empires is that they always have this narrative that there are good empires and evil empires in the world, and they, the good empire, have to be an empire in the first place.

To be strong enough to beat back the evil empire. And throughout my book I document how OpenAI has always had an evil empire, but they change who the evil empire is, depending on what’s convenient. So in the beginning, Google was the evil empire. Now increasingly, China is the evil empire, and this idea is that if the evil empire gets hold of this technology first, then humanity will go to hell.

But if they, the good empire, have unfettered access to resources, to labor, and they can get this technology first, then they will be able to civilize the world, bring progress and modernity to all of humanity. And humanity will ultimately have a chance to go to heaven.

Copy LinkWhy power in AI remains concentrated among a narrow elite

EL KALIOUBY: Yeah. And that comes back to this strong aspect of narrative and storytelling and belief, right? You have to have this strong belief around this very important mission. One of the things that really struck me reading this book is the cast of characters in this empire.

Right. And it shouldn’t be surprising to me because I’ve been in this industry for so long, but it still was really jarring that most of the book, there was just a lack of diversity and it’s specifically a lack of diversity around women. Right? I just find that really dismaying, so I would love to hear your thoughts. And it’s not just at the CEO level, it’s really the people building and shaping these key technologies and making the key decisions too.

HAO: Yeah, this is absolutely, like OpenAI sits at the intersection of the AI research world and Silicon Valley. Both worlds are ones where women just do not rise to the top. In AI research, I think the last stat I saw was only 12% of AI researchers are women. And in Silicon Valley, I think the most recent stat was only 2% of founders that receive investments from VCs are women.

And so automatically you have these two deeply male dominated sectors or fields merged into OpenAI, which then ultimately reflects that challenge where women are not rising to the top. Part of it is that there’s a lot of hostility in these environments towards women.

And another part of it is that women self-select to leave because they don’t agree with the general premise of what OpenAI or these other companies are pursuing — this idea of artificial general intelligence. So I personally have a number of friends who are AI researchers that chose not to go into the industry, in part because they just think that this quest to recreate human intelligence with this idea of a zero sum game, winner takes all, is just not the right one.

And they observe that this kind of quest typically ends up being the most harmful for marginalized communities, including women. So I think there are a lot of intersecting reasons why ultimately it is men that predominantly shape this technology.

Copy LinkHow network effects help turn startups into empires

EL KALIOUBY: I was also struck by the importance of this idea of a network effect, right? So for example, Sam Altman is an investor in many of the companies across the AI tech stack. Everything from nuclear fusion to health companies. And they’re always, like, it’s a group of people, they’re always supporting each other, investing in each other’s companies. Is that kind of an important aspect of empire building as well?

HAO: Absolutely. This idea of network effects is something that Altman learned from his mentors — that in order to aim for monopoly and create a dominant position in the market for anything that you’re building, you want to interlock both the people that you know and the investments that you know, so that you can continue to gain more and more leverage in that marketplace.

And so this is exactly the playbook that Altman then ends up using. He says many times throughout his career that one of the best pieces of advice that he got early on was this idea of building network effects. He is intentionally trying to interweave both his network of people and his network of companies to be as tightly integrated as possible so that it turns into a fortress, into an empire that is impenetrable.

EL KALIOUBY: In a minute, we break down the AI empire. We dig into what data extraction and labor exploitation actually looks like on the ground. And get into how we can build AI in a different, more ethical way. Stay with us.

[AD BREAK]

Copy LinkWhat data extraction really looks like behind generative AI

EL KALIOUBY: So I wanna kind of dig into this idea of extractive AI and this key element of empire building, which is extracting resources, whether it’s natural resources, or labor or data. Right? And I love this line. Data is the last frontier of colonization. And I think at this point we all know that LLMs are very data hungry and kind of often scraped off the internet, but let’s dig into that a little bit more. Like what were some of the key lessons you learned about how these companies are approaching access, not just to the quantity, but also to the quality of the data.

HAO: Yeah. And that quote was from Keoni Maona, who is an indigenous researcher and journalist and jack of all trades person that works for the nonprofit Te Hiku Media in New Zealand. And he was saying this to me as, like, as an indigenous person, this is so blatantly obvious that before, colonizers used to take our land and then sell it back to us. And now they’re just taking our data and turning it into a service and selling it back to us. And there was never any consent along the way. So originally, before OpenAI really started to dominate the scene, the AI research field and the industry was actually shifting more towards tiny AI and this idea of curating data sets and really making sure that it’s clean, pristine data that you’re using to train a model.

And ultimately that means you can get away with really, really small data sets that produce quite powerful models and also very predictable models in terms of their behavior, because you know exactly what you’re feeding into the system. But what OpenAI did was they started going for this large scale scraping of the internet.

And when you put everything from the internet into your training data, then you also get all of the bad stuff. So there’s plenty of gunk that gets left in, and that is part of what leads to downstream labor exploitation. And I interviewed this one executive of the company Appen, which is a third party platform that connects companies like OpenAI with contractors, workers in the global south or in economically vulnerable communities, that do data preparation and data cleaning. And he said before, we used to clean the inputs and now we put everything in and we control the outputs. That’s been the paradigm shift of the last few years. And what controlling the outputs ultimately means is that there are workers who have to perform the grotesque work of content moderation, because when you have a text generation model that can spew anything and is trained on the worst parts of the internet, it is going to start spewing really toxic, hateful content. And so to block that from ever reaching users, you need to wrap these models with filters that prevent this content from being exposed to the user. And that has downstream ramifications for the people that do that work.

Copy LinkThe hidden human cost of training and moderating AI

EL KALIOUBY: One of the stories that really struck me in the book is the story of Karina. Can you share that with us? Because I think it really drives home the labor that goes on behind the scenes in terms of building AI.

HAO: Yeah, so Karina was someone that I met actually pre generative AI era, when the AI industry, of course, already was relying very heavily on data annotation and data preparation. And she was a Venezuelan refugee that lived in Colombia. And the reason I went to go meet her is because Venezuela specifically became this huge hotbed of data annotators.

In the 2016 to 2020 era, because the country was undergoing the worst peacetime economic crisis in 50 years, at exactly the same moment that the self-driving car industry was taking off and suddenly needed tons of workers, cheap labor to do the annotation of self-driving cars to teach self-driving cars how to navigate the road. And what I learned was that structurally the data annotation industry has been designed to be exploitative. So she was working for Appen. And the way that Appen works is you can create an account and then when you log in, you have a stream of jobs that are available for you that are posted by companies, that just tell you to do tasks that you don’t know what they’re for.

And when she first joined the platform, this was generally a good premise. There were always tasks in her queue. She was able to take them. She was able to get paid several hundred dollars a month, which was enough for her to actually support herself, support her family in Colombia.

But there was an influx of workers and not enough jobs to go around. So by the time I met her, she was sometimes waiting four weeks at a time for a single task to appear. And that task could end up paying her just a few dollars.

And the problem is she never knew when that task would arrive. And so there was one day that she was on a walk outside when a task suddenly arrived and she sprinted back to her apartment to try and claim the task in time, because these platforms pit workers against each other. So you have to claim it.

And by the time she got back to her apartment, the task had already been claimed by someone else. It disappeared from her queue and she decided at that point to never go on a walk outside ever again.

During the weekday, she learned that during the weekends she could maybe get away with going out for a little bit. So she would limit herself to only a 30 minute walk on the weekends. It just made her life really small, because she wasn’t able to actually have freedom anymore.

EL KALIOUBY: I wanna share an example. So at my company, Affectiva, we had a data annotation team based in Cairo, which is where I’m originally from. And they were mostly women. And I’m actually very proud of the team. They’re still there even after we sold the company.

And as I was reading your examples of the stories of these data annotators, we actually decided to hire our data annotators as full-time employees. So they worked for the company, they had healthcare benefits, they had set time shifts. Right. So there is an alternative, more humane way of doing that.

HAO: Absolutely. This is something that I also talk about in the book — you know, this could have actually been the primary opportunity for the AI industry to do what they long pay lip service to, which is the idea that they’re gonna redistribute the economic benefits that they concentrate within their hands.

And if you think about it, what better way to distribute economic benefits than to properly create these professionalized jobs for data annotation, which is a key component of the AI development supply chain, and make it into a dignified economic opportunity. And the elements that you described of what you ensured at Affectiva is also what researchers have long advocated for in the digital labor rights community, which is you give them full-time jobs, you give them benefits, you tell them who they’re working for and why, you give them real managers that they can talk to, so that if they are experiencing some kind of adverse effect from the job, they can actually contest that and raise awareness. But unfortunately, most companies choose not to follow that guidance at all, and it becomes a race to the bottom for how little you can get away with paying these workers.

EL KALIOUBY: Yeah. There’s also the mental health side effects of some of these jobs because as you said, the input data to these LLMs is literally everything and anything that’s on the internet. And I was struck by some of the stories of the data annotators looking through sexual content and violent content.

HAO: Yeah, so Karina was pre generative AI era, but then what’s happened in the generative AI era is that there are still plenty of workers that are doing this work, but now the work has shifted to be problematic in and of itself. The content itself is dark and troubling, and so OpenAI at one point contracted these workers in Kenya to develop a content moderation filter, and I ended up going to speak with several of them, and I highlight the story of one man, Mophat Okinyi, who was on the sexual content team, where what he was expected to do was day in and day out read the worst sexual content on the internet, as well as AI generated content where OpenAI was prompting its own models to imagine the worst content on the internet, including sexual content. And he then had to categorize it into a detailed taxonomy of is this sexual fantasies or is it sexual abuse, is it sexual abuse that involves minors.

And they all had a different tag of severity that he had to assign to them. So Mophat ended up suffering the same fate that a lot of content moderators do in social media where his personality just fundamentally changed — he was extroverted and became very introverted and anxious, and when he would go home, he just could not engage anymore with his wife or with his stepdaughter, who he loved and called his baby girl. And he also couldn’t explain why, because he couldn’t say to them, oh, my job involves reading sexual content all day. It sounded really shameful. And so one day his wife asks him, I would like fish for dinner.

So he goes to the store and buys three fish, one for him, one for her, one for the stepdaughter. And by the time he comes home, all their bags are packed and they’re gone. And the wife just texts him, I don’t understand the man you’ve become anymore. We’re not coming back.

I didn’t put this in the book, but right after I came out of the interview, we walked into the hallway outside of his apartment and there was a neighbor’s baby girl that was crawling around the hallway and he just scooped up this little baby girl and was dotting on her and cooing to her and tossing her in the air.

And I went back to my hotel and cried. I was like, I cannot believe that we are allowing these people to suffer in this way, where he has lost his baby girl, and now all he can do is play with his neighbor’s baby girl. That’s just so gut wrenching.

EL KALIOUBY: Sad. Just in the news, just as we’re having this interview, Scale AI announced that they’re getting a multi-billion dollar investment from Meta and Alexander Wang, the CEO, is joining Meta to lead AI efforts there. But OpenAI, which used Scale AI as a data provider, just announced that they’re dropping the company as a data provider. Did you see that coming?

HAO: I did not, but it makes a lot of sense on Meta’s part to do something like that because Scale became the go-to platform for this kind of data annotation work, and it would make sense for a company to try and then buy it all up because Scale had visibility into the model development practices of all of the major players. Purely as a business perspective, it is a very clever move to essentially — they’re not just acquiring the platform, they’re acquiring the knowledge that that platform accumulated on all of the different contracts that they had. And of course, now the other companies feel that they can’t use this service anymore because then it’ll just be a straight funnel of information directly to Meta.

And so they’re also taking one of the major players out of the market for other people to use. Scale has also been riddled with labor exploitation. That is one aspect of the story that really hasn’t been surfacing in the headlines of this major acquisition.

EL KALIOUBY: AI can be extractive and exploitative, as Karen’s reporting shows. And while data scraping and unfair labor practices aren’t often making headlines around AI … you know what is? The race towards the elusive goal of AGI. More on that in a minute. Stay with us.

[AD BREAK]

Copy LinkWhy OpenAI's mission became harder to define and defend

EL KALIOUBY: So let’s switch focus to talking about OpenAI’s mission, which as you said, is to ensure artificial general intelligence benefits all of humanity. You argue in the book very clearly and convincingly that OpenAI has kind of veered off from this original mission. How so?

HAO: Well, it’s interesting because they’ve never really been able to define their mission. Early on when I started profiling the company, what I quickly realized was there was no consensus on what “ensure” means. There was no consensus on what AGI means, and there was no consensus on what “benefits all of humanity” means.

Whether or not they veered off their mission is hard to say because what was the mission in the first place? But OpenAI has shifted its interpretation — whereas before it interpreted the mission to mean we are gonna be a transparent, collaborative nonprofit, now they’ve reinterpreted it to mean we’re going to be a product focused, deeply commercial, one of the most capitalistic companies in the world, to deliver this technology into the hands of everyone, but make sure that we are the conduit through which everyone accesses that technology. And so that has been one of the most dramatic about-faces that has ever been seen in Silicon Valley, but all apparently in the name of the same mission.

EL KALIOUBY: Yeah, so allegedly Sam Altman had a call with President Trump where he shared that he thinks AGI will happen during this presidency. Do you agree?

HAO: So OpenAI has long had this joke that if you ask 13 employees what AGI is, you’ll get 15 definitions. So depending on what definition Altman is using, if they’re using the specific one that they’ve written down as a company, the labor automating definition, then I could see there being a world in which within the Trump administration there will be significant job loss.

And I wanna highlight that it’s not because the AI models will become so hugely capable that it will lead to this job loss, but that they will be persuasive enough to convince executives to lay off workers. In fact, we are already seeing that — there are already economic indicators coming out that entry level jobs are disappearing and that new graduates are facing job crises.

Copy LinkWhy human augmentation is a better path than replacing people

EL KALIOUBY: My whole career in AI has been centered around building human-centric AI. So AI that augments and amplifies human ability and unlocks human potential. And I really have this conviction that we can still create massive economic opportunity while also doing this in a very human centered way. Do you think the way we’re approaching AGI is human centered?

HAO: No, because the premise of AGI is the idea that we replace humans. And I cite a book in my book, Power and Progress by Daron Acemoglu and Simon Johnson, two MIT economists that won the Nobel Prize last year. And in their book they talk about how ultimately, if you want to unlock the benefits of AI, you need to design it as human assisting, as you mentioned, human augmenting.

And those are two fundamentally different design approaches. The industry has consistently taken the human automating approach where ultimately the strengths of humans are trying to be replicated instead of the weaknesses of humans being addressed. And that is one of the most baffling things about the current direction of AI development, in that humans already are good at what humans are good at. Like, why are we trying to build machines to take over what humans are good at? Why not try and build machines to take over what humans are bad at so that we can work hand in hand together? And there’s been plenty of research to show, for example, if you have a cancer detection AI system, which is not AGI, it’s a very task specific AI tool, placed in the hands of a well-trained radiologist or cancer specialist, then you will have the ability to diagnose cancer earlier and more accurately than the doctor or the AI system could have done alone. And that is the model that we should be going for. But unfortunately, we are just not seeing an emphasis on this approach.

Copy LinkHow to push AI toward a more democratic and ethical future

EL KALIOUBY: Okay, so let’s talk about what the future looks like and you end the book with some thoughts on how this empire potentially falls.

HAO: So I very much feel not only was the particular path that was chosen today not inevitable. Our future is also not inevitable. Allowing these empires to just continue proliferating completely unchecked is not inevitable. And ultimately it really comes down to the fact that empires are built off of an overlapping monopoly of power on many, many different axes. And so one of the ways that I see us redistributing power and containing the empire is by looking at all of those different power monopolies and figuring out ways to pull their monopoly away.

So how can, for example, government agencies pump more research funding into the public domain to create independent institutions of AI expertise? That is one way that we can start having knowledge production outside the empire so that we can see whether or not these technologies do have limitations and ask scientific questions that are inconvenient to these companies, such as can we actually just train extremely data efficient and computationally efficient models? And the answer is yes, we can. We just need more investment into that area. I also think that it’s not just top down government agencies that can do this work, but there are literally anyone listening to this podcast, anyone living in the world, has agency in shaping.

The future of AI development, and that everyone intersects in multiple ways with what I consider to be the full AI development supply chain. When you think about all of the different ingredients that these companies ultimately need to make their technologies, they need the data, they need the land.

They need energy to power their data centers. They need fresh water to cool their data centers. They then need access to all these spaces to deploy their technologies like schools, hospitals, businesses, government agencies. These resources and these spaces are actually collectively owned and collectively governed. And I see them as sites of democratic contestation. So we are already seeing artists and writers suing these companies saying, no, you can’t take our intellectual property. And that is them playing an active role and reclaiming agency and ownership over a critical resource that these companies need, and that is forcing companies to actually start thinking, are there different models in which we do actually provide credit and compensation for accessing this data?

If you are using these tools, you are giving that data to those companies. But if you refrain from using the tools, then that is you withholding, reclaiming ownership over your data from those companies. Your social media presence — I actually ended up deleting all of my personal social media accounts.

I still have professional ones, but I removed all of my personal photos. I removed all my personal thoughts and ideas and things as companies started changing their policies to train their generative AI models on these, and that was me reclaiming ownership over that. And we are seeing hundreds of communities push back on data center development that’s happening in their communities in mutually unbeneficial ways. So I talk about these Chilean water activists that had this amazing spirit of resistance where when a data center from Google was being proposed in their community and proposed to use all of the fresh water in their community, they put their foot down and said, no, what are you going to give us in return?

And they successfully have stalled that project for five years and counting now because the company still has not put down a proposal that they find to be mutually agreeable. We are seeing teachers and students actually talk about, wait a minute, do we want AI in our school systems? And if so, under what conditions and what kind of AI? And so all of these discussions, if we can have them a hundred thousand times over, I think it will force the companies to shift their practices, ultimately away from an imperial AI development approach to a much more democratic one.

EL KALIOUBY: Yeah. I was also struck by, and again being an entrepreneur myself, I knew that, but we often forget that all these technologies we use are a result of a small group of people sitting somewhere making big or little decisions day in and day out, right? And it so happens that you’ve uncovered the stories of how AI gets made, but how do we bring more people around the table to ensure that there are diverse perspectives and voices in how we build this thing that’s gonna affect all of us.

HAO: To anyone that’s listening, that is an entrepreneur, an investor: you absolutely have significant agency in changing the direction of AI development. Part of the challenge right now is consumers of these tools do not have many options, and we just need more options for ethically sourced and sustainable AI development.

Ultimately we need more AI systems that are small, data efficient, computationally efficient, targeting a specific challenge that AI is good at, that is computational in nature. And in fact, I think that there’s a huge market opportunity that is not being tapped into right now, both for entrepreneurs and for investors, and if you are an early mover in this space, you could be defining the next revolution in AI.

EL KALIOUBY: I love that call to action. What a great note to end on. Thank you, Karen, for joining us. This was great.

HAO: Thank you so much, Ronna.

EL KALIOUBY: We covered so much in this conversation, but there’s even more in Karen’s book. “Empire of AI” dives deep into OpenAI’s internal power struggles, its spinoffs, and its fierce competitors. It’s a fascinating read, which I highly recommend.

In Karen’s mind – and in my mind, too – we do have agency when it comes to the future of AI. We can shape AI away from empires … (that just extract data, and exploit humans and natural resources) … toward a different distribution of power … (one that’s more decentralized and human-centric).