Blocking AI crawlers to save the internet

Matthew Prince: We go to war every day with Russian hackers, Iranian hackers, North Korean hackers, Chinese hackers, who are trying to get in and thwart our systems and get around it in various ways. The AI companies are a piece of cake in terms of blocking versus the real hackers that are out there.

Bob Safian: That’s Matthew Prince, CEO of Cloudflare, the $60 billion tech firm that facilitates more than 20% of all online traffic. Cloudflare just released a buzzy new tool that blocks AI crawlers from scraping content from websites. I wanted to talk to Matthew because Cloudflare’s tool could have dramatic impact on AI firms, on news and content sites, and on the future of the internet. Plus, he has an insider’s view on the state of cyber warfare right now in the age of AI. So, let’s get to it. I’m Bob Safian, and this is Rapid Response. 

[THEME MUSIC]

I’m Bob Safian, I’m here with Matthew Prince, CEO of Cloudflare. Matthew, thanks for being here, thanks for joining us.

PRINCE: Really appreciate you having me on.

SAFIAN: So, I have to say, what Cloudflare does, an internet infrastructure provider, it isn’t always seen as the sexiest tech, but you do touch more than 20% of all online activity, right? You stand between website visitors and a website, or an app itself, managing traffic and content, so the experience is smoother and faster and more secure. But recently you released a new tool that’s got a lot of folks buzzing, a blocker for AI crawlers, the bots that scrape content from websites without their consent. You’ve called this new tool the biggest thing you or the company has ever accomplished, you actually gathered your team atop the World Trade Center, and pushed a big button to launch it?

PRINCE: Yeah. So, I feel incredibly fortunate to have built what today is a $60 billion company on the back of the internet. And we became aware about 18 months ago of a new threat, and we’ll, I’m sure, talk about that. But to the internet, to the web, to content creators, which was being posed by these AI companies, I don’t think intentionally, I think most of them are actually trying to do the right thing, and out for preserving the ecosystem, but they created this real challenge. And so, when we realized that there was something we could do about it, we spent about a year talking to everyone in the content creation space, everyone in the AI space, we actually signed a lease on a new office space in New York, which is the top occupied floor in the World Trade Center, and we hosted a party, and at midnight we pushed a big red button.

It was me, along with really the leaders in the publishing space, who recognized that the challenge that was there, every publisher from the Associated Press to Ziff Davis, and everything in between, we’re going to change the rules of the road, and say that if you’re not paying for content as an AI company, then you don’t get that content.

SAFIAN: And the threat that you saw, that a lot of folks like me in journalism and media see, it was that the old bargain that you could crawl my site and then you’d send me traffic back, basically, that the AI companies, that’s not really what their model is, right?

Copy LinkThe shifting value exchange between content and AI

PRINCE: The original deal that Google made with content creators was, give us your content and then in exchange we’ll send you traffic, and you can create value with that traffic. Either you can sell something, the content itself, or a subscription to the content, or an e-commerce business that’s on the side, you can sell ads against it. You can get value from just the ego of knowing that someone is consuming what you created, no one gets paid for editing… And we have really good data going back, at least 10 years, on sort of how that bargain has changed. If you just look at Google, you saw it get about three times harder to actually generate traffic from the same sort of content that you created because Google changed the user interface, they added an answer box at the top of the page. So, if you type in, “When did Cloudflare launch?” Instead of you having to click on one of the links to find the answer, it answered right on the page.

And again, it’s better for users. We shouldn’t dispute that this is better for the Google user getting the answer right there, than having to sort through documents. But it meant the answer box made it three times harder to get traffic. They then, in the last six months, added AI overviews, and the AI overviews took the answer box and they essentially put it on steroids. And so, today it’s almost 10 times harder to get that actual traffic from Google for the same amount of content you created. And this isn’t just Cloudflare data, Pew Research just came out with a study that lines up exactly the same, and says that the minute you show an AI overview, it’s less likely that people click on links. And again, that is better for the Google user, but it is worse for the content creator because it means that you can’t sell a subscription, you can’t sell ads, and you can’t even get the ego boost of knowing that people are reading your stuff.

And that’s the good news. Google is trending towards something that looks more like OpenAI. Today, OpenAI is 750 times harder to get traffic from than Google of old. Anthropic is 30,000 harder to get traffic from than the Google of old. And so, if publishing, if content creation is struggling today at it being 10 times harder, I worry that it won’t survive at 750 or 30,000 times harder, where instead of reading original content or reading derivatives of that content, and if people don’t have the incentive to create content, they’re not going to create content. Reporters need to eat, journalists need to eat, academics need to eat, researchers need to eat… So, there needs to be some business model, or at least some value creation model behind the future of the web, and it’s not going to be around traffic because an AI driven web doesn’t drive traffic.

SAFIAN: Well, and the irony is that the AI itself needs the content that folks are creating to be able to make those answers, and now, who knows where they’re going to get their answers from?

PRINCE: That’s the key. 80% of the major AI companies use Cloudflare in their infrastructure. So, we know them pretty well, and we have had over the course of the last several months, conversations about this. And then what they have all said, with a few exceptions, but almost all, is we agree, content creators need to get paid for content, but it has to be a level playing field. What nobody wants to do is that they pay for content where all of their competitors get it for free. Or the flip side, where somebody gets it for free and everybody else has to pay. Because what you ideally want, you want a lot of people creating content, but you also want a ton of different buyers. So, you want to make sure that new startups can participate, established players that have been in some other space aren’t at a disadvantage to someone like Google, who’s been able to crawl the internet with impunity for an enormous amount of time, you want to make sure that someone who’s hyper-regulated like Google can still compete with new these startups.

So, creating that level playing field is incredibly important, and ensuring that there’s a healthy ecosystem, if we can figure out how to do that, if we can figure out how to say, listen, everyone has to pay, the rules are the same and fair across the board for everyone, I’m encouraged that the AI companies have actually said yes, and we’re on board. Because ultimately we know that the fuel that fuels our engines, the thing that is critical is that original content that’s out there. Just Anthropic will scrape a site 60,000 times for every one visitor that’s there. And what’s key is that’s real cost, right? Someone has to pay for that traffic. And so, just from a pure fairness perspective, they should be compensating creators that they’re pulling that content from.

Copy LinkChallenges and complexities of blocking AI crawlers

SAFIAN: And this blocker tool that you came out with, is the tech itself hard fundamentally? I’m sure it’s not easy, but how much of it is about the tech versus the strategic that you guys made to, all right, we’re going to challenge this, which seemingly no one else was trying to do?

PRINCE: It’s hard to say what’s hard anymore. We process something like 20 trillion internet requests a day, which is just an inconceivable number. So, just a mere fact that we do that is hard, and if you think about who we go to war with, what our primary business is, we started as a cybersecurity company. And so, we go to war every day with Russian hackers, Iranian hackers, North Korean hackers, Chinese hackers, who are trying to get in and thwart our systems and get around it in various ways. And so, it was kind of funny, when we first started talking to publishers about this, it was almost this sort of nihilistic kind of, oh my gosh, what are we possibly going to do? There’s no way we can stop it. These guys are so smart, they’re a bunch of nerds, they’re in Palo Alto… We can’t ever possibly block them. And I remember thinking, we block the North Koreans every day, and they’re not subject to the Computer Fraud and Abuse Act, and they don’t have a C corporation that you can sue.

The AI companies are a piece of cake in terms of blocking versus the real hackers that are out there. That said, I think the hard thing isn’t the blocking, the hard thing is making sure that you’re doing it in a way that creates a level playing field. Google is a huge challenge, and here’s why. Google has one bot, and that bot does five separate things. It does brand safety checks, so if you’re Procter and Gamble, you don’t want your ads being run against pornographic content, or something like that, and so the bot checks a page to make sure it’s not porn, and then only releases a Procter and Gamble ad if it’s not. So, it does that. It does traditional search indexing, it does data mining for the answer box, it does data aggregation and collection for AI overviews, and it does training for Gemini.

SAFIAN: So, one bot does all of these things.

PRINCE: Five things.

SAFIAN: It’s all in one.

PRINCE: Yeah. And they give you three different options today. They can say, you block everything, but that sucks because now you’re not in Google search results, now you’re not able to put any ads on your pages because Google powers so much of the ad ecosystem. You can allow everything, but that sucks because now your content is being taken from you, and you’re not getting any value exchange for those three things, for answer box, AI overviews, or Gemini. Or you can block Gemini, the fifth thing. They’ve actually given you the ability to block that one thing and not block the other four. What we think is we need to create really a taxonomy, and an industry standard on what responsible crawling is, and that’s super hard. But we’re working with standards organizations like the ITF to set down what those standards are.

And we’re very hopeful, because if Google is the hard case, on the other side, who’s doing this really responsibly is OpenAI. OpenAI says, okay, we have one bot for training, we have a different bot for search, we distinguish between those things, we publish clearly who our bot is, they’ve been willing to go into actually cryptographically signing that their bot is their bot, so that we can tell that someone’s not forging it or something else, they follow the equivalent of the street signs that are out there, and they crawl at a reasonable rate. And again, I think that that’s a very good example of a company that is trying to do the right thing.

SAFIAN: And before you release this first round of this tool, because clearly you’re continuing to develop it, did you give the AI companies a heads-up, or some of them, some of them are clients, but I can imagine that there are some AI companies, they’re like, oh, how do we get around this new tool?

Copy LinkAddressing good and bad actors in the AI space

PRINCE: I think there are some bad actors out there, and I think it’ll surprise some people who the bad actors are. Some of them won’t be surprises. But there are some folks who basically, if they can’t get through one way, they try a bunch of other things that start to resemble what the North Koreans do. Now we again, are really good at this, so we’re tracking them, we’re monitoring them, and very soon we will publish and we will name and shame who is actually a bad actor in this space. And we will take from what has been basically posting a speed limit sign, that says, don’t drive more than 55 miles an hour, we’ll take that speed limit sign that we have, that we put up on July 1st, and we’ll make it into something that is actually much more strict, where we’re saying, listen, we’re taking away your car, you’re not allowed to drive on the road anymore.

At the same time that we do that, I think we’re going to highlight some of the companies that are doing it right, because you want to actually say, these are companies that are good actors that are out there. And so, as we start to separate the good actors from the bad actors, and by far most of the AI companies are actually good actors, there are a handful that are doing some pretty shady stuff, and again, we’ll shut it down and hopefully they’ll reform.

SAFIAN: I understand you’re exploring sort of a pay-per-crawl model with some of the content publishers, which to me sounds a little bit like a toll on the highway there, that you have to pay a toll if you want to come through.

Copy LinkExploring new business models for content compensation

PRINCE: Well, again, just simple first principles. If you are generating a huge amount of cost by crawling somebody, but you’re not giving them any benefit, then step one is block them. But if that’s valuable to you, then once you’ve created scarcity, then there can be a market, right? There’s already a lot of content that AI companies are paying for. OpenAI, again, is one of the more responsible players in the industry from everything that we’ve observed, has done deals with a lot of content providers, saying, we’ll pay for the content. But Sam can’t be a sucker, he can’t be like, I’m paying for it, but all my competitors get it for free. And so, I think that comes back to why the reception from the AI companies was actually a lot more positive than people might assume, because what everyone wants… Everyone agrees the ecosystem has to be healthy, again, almost everyone agrees, a couple of bad actors. But almost everyone agrees the ecosystem has to be healthy.

There has to be some compensation for taking content, and it’s not going to be traffic anymore, it’s going to be something else. Now, the question is, okay, how do you pay? And I think a lot of times, big AI companies and big publishers are just going to negotiate deals themselves. So, if you’re Condé Nast, you go out and do an OpenAI deal, or a Google deal, or something else, and you negotiate it yourself. We don’t have, again, we can provide the blocking technology, but we don’t have any role in that. I think for the smaller AI companies, or for the long tail of publishers, that Cloudflare can hopefully sit in between and help negotiate what is the best deal. And we don’t know exactly what that will look like yet.

It could be a micropayment every time a page is accessed, it could be something that’s closer to a Spotify, model where there’s a pool of funds and that gets distributed out all to all of the different content providers, I don’t know, that will develop, but step one in any market has to be scarcity. If you don’t have scarcity, you don’t have a market.

SAFIAN: And the bad actors that you refer to, I know you’re not prepared yet to name them, but are these companies that are non-US companies, in China and other places, or are these —

PRINCE: No, we have some very bad actors that are venture-backed, very successful companies. And we have reached out to them directly and said, listen, we see you doing this, you got to stop or we’re going to shut down your access to the roads. And we know a lot of their investors, we reached out to their investors, and we’ve given them every possible chance. And so, now, we’re at the final stage. Next step is, name and shame them. And again, that’s not something that I’m excited about doing, but I do think that we’ve got to get… We have to establish this level playing field.

SAFIAN: Let’s just acknowledge that Matthew is really poking the bear here in challenging how AI companies operate, going after the hottest players in tech who are driving global markets and the global economy, but change is necessary, at least from my perspective in media, as a content creator. So, what will the internet business models of the future look like? We’ll talk about that after the break. Stay with us. 

[AD BREAK]

Before the break, Cloudflare’s Matthew Prince talked about a new tool to block AI companies from crawling content sites. Now, he talks about what a new business model for the internet might look like, how cyber warfare is shifting in the age of AI, plus his plea for fixing US immigration policy. We begin with Matthew’s take on where internet business models and incentives are moving to, and whether it’s likely to be as robust as what we’ve had in the past. Let’s jump back in.

Copy LinkHow AI could drive a golden age of content creation

PRINCE: I’m encouraged by analogous markets that have developed. Before Steve Jobs launches 99 cents per song with iTunes, the music industry was about an $8 billion industry, in terms of what their annual revenue was. If you just look at Spotify, last year, Spotify distributed $10 billion to musicians. And so, even though everyone thought, oh, digital is going to kill the music industry, it actually ended up being pie expanding. And so, I’m actually optimistic, all of us are going to have subscriptions to a certain number of AI agents that are out there, that revenue that goes to the AI company, some portion of it has to go to GPUs, some portion of it has to go to AI researchers, some portion of it has to go to your other overhead, but some portion of it should also go to content companies and content creators. And my prediction is that the algorithms and the GPUs will turn out to be commodities largely.

And how AI companies will differentiate themselves is access to unique content that they have and they have alone. So, imagine, Taylor Swift is about to release a new album, and she does an interview with some journalists, and they are willing to give that interview to one AI company exclusively for a week. How much is that worth? Probably quite a bit, right? A lot of people are going to sign up if you have it. It’s like, if Netflix is willing to spend tons of money to produce movies, there’s going to be value in actually creating unique content that your AI has that another AI doesn’t have. And so, I’m actually optimistic that we might be at the precipice of a golden age of content creation.

If we do this right, and we get the incentives right, it might be that instead of us all worshiping the deity that Google taught us to worship, which is traffic, which has always been a really bad proxy for value, if instead we find a way to compensate creators based on when they actually create something which is worthwhile and advances human knowledge, that we can actually do some real good in the world, at the same time that we help the content creators get paid more.

SAFIAN: I hope you’re right, Matthew. I sure do. It does sound like the world I want to live in.

PRINCE: Think about everything that’s wrong with the world today, and again, I don’t care whether you’re a Democrat or Republican, or conservative or liberal. No matter what, we’ve gotten more polarized, we’ve gotten more anxious, everything feels urgent and panicky. And if you ask, why is that the case? This is going to be too strong a statement, and I risk pissing my friends at Google off, but sort of it’s all Google’s fault. I think Google has been a net positive for the universe, I think they’re good, but they did teach us all to worship this deity of traffic. And the question is, how do you get traffic? And so, there was a whole series of companies that came along, the Huffington Post, and BuzzFeeds, and others of the world, where they would create a piece of content and they would literally beta test different headlines for which ones created the largest cortisol response, the largest stress response in readers, because that was the thing that was most likely to get a click.

I don’t know that that’s good for the world. And again, Google just started it, Facebook took it to the next level, TikTok is now taking it to a whole other level, but that world in which we are just trying to stimulate a cortisol response, and that’s what success in media is, I think is really, really, really dangerous. If, on the other hand, you think about all of the AI companies that are out there, it’s not a perfect representation, but it’s a pretty good representation of a map of what human knowledge looks like. And I picture it like a big block of Swiss cheese, right? So, big block of Swiss cheese, lots of cheese, but lots of holes in that cheese. If we could actually help AI companies say, okay… And this is actually a technically solvable problem. When a new piece of content comes out, if we say, hey, does this just add more cheese to where we are, or does this help fill a hole?

And we can say that you get compensated more by filling those holes, I think that’s actually the way that you advance human knowledge. Again, I think Spotify has done some just amazing things. One of the things they do is they actually publish a list of queries that people have typed into Spotify, of music they want to hear, that they don’t give very satisfactory answers to. So, let’s say that you want to hear, I don’t know, a disco about how much fun it is to dance with your cat. There’s no songs out there like that. They publish that list, and there are music creators out there that look at that list and they write songs based on what people are searching for, this unmet need in the market. And they make tens of millions of dollars a year writing songs based on unsatisfactorily searched Spotify queries.

If we can get to that place where, again, instead of us thinking, how do we write content, which is yet another story about what crazy things are happening in politics, or how… Whatever it is that makes people just get up in arms and click on things. If instead we can say, what is it where there’s an unmet human need to advance human knowledge, and we can reward content creators for that, I think that could again be the golden age of content creation.

Copy LinkThe evolving landscape of cyber warfare and AI threats

SAFIAN: You mentioned earlier that your heritage at Cloudflare is from cybersecurity, I want to ask you about the topic of cyber warfare. Wars today, they emerge in cyberspace before they’re on the battlefield, we’ve seen that in Ukraine… Your detection tools have been utilized in a lot of these situations, right? You’re a partner in these cyber battles?

PRINCE: Yeah, I think we’re a defensive partner, we don’t make bullets, but we make shields. And we see things that are indicative of attacks. So, back in December of 2021, we started to see the indicia of what Russia does before they launch attacks. And unfortunately, we have experience with that with Georgia, with Crimea, with Syria, and so we know what their playbook is, and so we can start to see, oh, they’re probing in this way. And I felt a little bit like chicken little, where, for a while we were running around being like, the Russians are about to attack Ukraine, the Russians are going to attack Ukraine, and a lot of government officials were like, no, they’re not. But thankfully, we were able to get connected with the Ukrainians, we provided our services at no cost, to a bunch of their critical infrastructure, and when the Russians did attack in February of 2022, we were able to keep a lot of the Ukrainian internet online.

And again, we were one small player in a huge effort. I thought that the next move from Russia after attacking Ukraine would be to go after their Western allies. And there was a little of that, but nothing near what we expected, which would’ve been a much larger and broader attack. It actually wasn’t, Russia’s cyber attacks didn’t really pick up until something that seems completely unrelated happened, which was the Hamas attack on Israel. When that happened, all of a sudden you saw a whole bunch of Russian attacks, that were trying to look like they were actually Hamas attacks.

They were hiding behind that. And I think the reason for that is, every country is enormously vulnerable to cyber attacks. Russia’s entire power grid could be knocked offline by the US, if they wanted to. And so, there needs to be, in the cyber war, a little bit of plausible deniability, and so it required this other war to start before Russia could launder their attacks through that. And so, we’ve definitely seen an increase in Russian attacks, we can attribute it back, but they’re making some effort to make it look like it’s coming out of these other places.

SAFIAN: The rise of AI tools, is that making cybersecurity tougher or is it just mostly showing up in the robocalls on my phone, and the spam emails I’m seeing in my inbox, or is it everywhere?

PRINCE: The attackers are getting better because of the tools, but the defenders are getting better faster because of the tools. Machine learning algorithms, they were pretty good at finding things we already knew about, but they weren’t good at identifying new threats that no one had ever seen before. Starting about two and a half years ago, they started identifying threats that no human had ever identified before, that’s now something that’s happening on a very regular basis internally at Cloudflare. That doesn’t mean that there aren’t going to be horrible things that happen because of AI, but on balance, I think the internet is safer because of AI, not less safe.

Copy LinkThe impact of politics on American tech companies

SAFIAN: So, Cloudflare, you started as a project at Harvard Business School. Now, how do you feel about the troubles Harvard’s having with the Trump administration? Should Harvard be expecting successful alums like you to donate and fill their research in their funding gaps?

PRINCE: I have to confess, I’m not an expert in academia, and everything that’s going on. Again, Harvard was great to us. I will say that my co-founder, Michelle, Michelle’s Canadian. And back in the Obama administration, she had come to the US on a student visa, she was allowed to stay after graduation for a limited period of time, but we had to apply for her to be able to stay in the country. And it was really, really hard. And she came this close getting kicked out of the country, and had that happened, Cloudflare wouldn’t exist. And so, I do think that as we think through these things, from my very micro perspective, making sure that we have paths for the Michelles of the world, who’ve created thousands and thousands and thousands of jobs directly in the United States, advanced enormous amounts of shareholder value, we’ve got to make sure that as we fight whatever the other wars are, whether cultural or political or whatever, that we make sure that folks like Michelle can still come here and succeed.

And I think that’s a superpower of the US. If the US wanted to completely and totally annihilate China, if we decided that that was our goal, and I’m not saying it is, the most effective thing that we could do is offer admission and scholarships, plus, essentially green cards, to the families of the top ranked students coming out of Chinese education, they would all flock the United States, they would all stay in the United States, and they would all create value in the United States. And because of how hierarchical the Chinese system is, you would eliminate a bunch of future leadership of the Chinese government. The fact that we make it difficult for really, really, really talented entrepreneurs or researchers or whatever to stay here and create value here, that worries me a lot.

SAFIAN: Philosophically, a lot of leaders today are struggling to envision the future in 2025, because so much is changing so fast. How much do you think about Cloudflare shaping the future versus having to adapt to what comes?

PRINCE: There was this almost two hour long interview with Steve Ballmer, and the thing that struck me the most was how even in 2000, when the Justice Department comes after Microsoft and everything, how they were still very much, thought of themselves as the little guy. Folks at Google will be like, oh my gosh, we can’t do that because we’d get crushed, I’m like, you’re Google, how are you possibly thinking that way? I think we still continue to fall into that, we don’t realize quite the impact that we could have on things. I think we’re always surprised, like when we did pay per crawl and it did feel like the industry shifted, I think internally we were actually quite surprised. There were plenty of times we do things and then it’s crickets. Back in 2020, we launched with this graphics chips company, an initiative where we would put GPUs at the edge of our network so that people could do AI inference, and nobody picked it up. It was complete total crickets.

The graphics chip company, I don’t know if you’ve heard of them, a company called Nvidia. Now, four years later, it turned out the market was ready for it, we literally just dusted off the same press release, issued it again, and we’re like, hey, by the way, we have GPUs at the edge of our network. So, again, I still think we very much feel like we’re adapting, but I actually think that that’s how Google feels, I think it’s how Microsoft feels, I think it’s how Meta feels… I think everybody, they are nervous about the decisions that they make, they don’t have nearly the degrees of freedom and control that you might think that they do, I think mostly we just kind of feel like the world is happening and we have to attack and adjust to whatever it does.

SAFIAN: Well, Matthew, this was great, thanks so much for doing it.

PRINCE: Absolutely. Thank you for having me on.

SAFIAN: When Matthew talks about Cloudflare, and Google, and Meta and others hustling to adapt to the pace of change, it’s a reminder to the rest of us that today’s chaotic environment isn’t easy for anyone. At the same time, at least when it comes to blocking AI crawlers, few companies are pushing to consciously shape the future the way Cloudflare seemingly is right now. I’m struck by how Matthew and his team are using case studies from the likes of iTunes and Spotify to find models for the next chapter of the internet. The past may not repeat itself, but getting inspiration from previous challenges can provide valuable inspiration. You’ve got to dream it to build it, and Cloudflare is definitely dreaming big. I’m Bob Safian, thanks for listening.