The New York Times Blocks ChatGPT, Claude and Perplexity. Google Still Gets In.
The New York Times robots.txt names 61 bots and blocks every AI crawler, but not Googlebot. What it tells a small company about its own.
In my last post I checked 50 SaaS websites. None of them blocked the AI answer bots.
So I wanted to see the opposite: a site that blocks almost everything on purpose. The New York Times was the obvious pick.
I ran nytimes.com through my free robots.txt checker and then read the whole file myself. Here’s what’s in it.
What robots.txt is (30 seconds)
robots.txt is a small public text file at the root of every website (for example nytimes.com/robots.txt). It tells bots which pages they may visit.
Each bot has a name, like GPTBot or Googlebot. The file can say “this bot, stay out” or “this bot, come in”.
Polite bots follow it. It’s a request, not a lock.
1. 61 bots are blocked from the whole site
The file names 61 bots with Disallow: /, which means “don’t read any page”.
That includes the bots behind almost every AI assistant:
- OpenAI: GPTBot (training), OAI-SearchBot (ChatGPT search), ChatGPT-User (when a user asks ChatGPT to open a page)
- Anthropic: ClaudeBot, Claude-SearchBot, Claude-User
- Perplexity: PerplexityBot, Perplexity-User
- Google: Google-Extended, the setting for whether Gemini can train on a site and use it in answers
- Others: Applebot-Extended, Meta’s AI bots, DuckAssistBot, CCBot (Common Crawl), Bytespider and many more
The top of the file says why. A comment there says that using their content to build AI or large language models is not allowed without written permission.
2. Googlebot gets in
Googlebot is not blocked. It’s kept out of some areas, like ad pages, logins and checkout, but it can read the news.
That matters, because Google’s AI Overviews and AI Mode are part of Google Search, and Google Search is fed by Googlebot. Blocking Google-Extended doesn’t change that. Google’s own docs say Google-Extended “does not impact a site’s inclusion in Google Search”.
In August 2026 Google added a switch in Search Console that lets sites leave AI Overviews and AI Mode. I can’t see from outside if the Times uses it.
So the picture is: out of ChatGPT, Claude and Perplexity by robots.txt. Still open to Google Search.
3. Some search engines aren’t named at all
Bingbot and Applebot (Apple’s search crawler, used for Siri and Spotlight) aren’t named in the file. A bot that isn’t named follows the general rules for everyone, and those rules don’t block the whole site.
So these two can read most of nytimes.com.
This is a good lesson for any site: a bot you don’t name gets the default rules. If you want to be sure about a bot, name it.
4. Amazon is only kept out of one section
Amazonbot has its own line: Disallow: /wirecutter/. That’s it. The rest of the site is open to it.
That matches the news. In May 2025 the Times signed an AI licensing deal with Amazon that covers its news, Cooking and The Athletic. Wirecutter was left out of the deal.
So the file isn’t just “block AI”. It’s “block AI unless we have a deal”.
5. Blocking isn’t the whole story
robots.txt is only a request. Two things are worth knowing:
- OpenAI says its rules for ChatGPT-User “may not apply” in the same way, because a person asked for that page.
- Perplexity says Perplexity-User generally ignores robots.txt, for the same reason.
The Times is also suing OpenAI and Microsoft (since 2023) and Perplexity (since December 2025). For them, robots.txt is one tool among several.
What this means for a small company
The New York Times sells its writing. Its content is the product, so it wants to control who uses it and get paid for it.
You’re probably the opposite. You sell a product, and you want ChatGPT to mention it when someone asks “what’s the best tool for X?”
So don’t copy a publisher’s robots.txt. And check that yours doesn’t block AI answer bots by accident. It happens more than you’d think, often from an old template or a “block AI” setting someone turned on once.
If you do want to stay out of AI training but stay in AI answers, you can. Block the training bots (GPTBot, ClaudeBot, Google-Extended) and allow the answer bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot). The two are separate.
A quick note
- I checked the file on 13 November 2026. Big sites change it often.
- robots.txt shows what a site asks for. It doesn’t show deals, firewalls or what each bot really does.
Check your own
The checker I used is free. It reads your robots.txt and shows which AI bots can read your site and which can’t, and gives you the lines to add if something is blocked by mistake: dowseai.com/robots-txt-checker
Being readable is the first step. The next question is what ChatGPT, Claude, Gemini and Perplexity actually say when buyers ask about your category. I built DowseAI to show that: who they recommend, what they get wrong about you, and what to fix. It’s $29, once. (dowseai.com)
FAQ
Does blocking GPTBot remove me from ChatGPT answers? Not by itself. GPTBot is for training. ChatGPT search uses OAI-SearchBot. They’re separate, and you can allow one and block the other.
Does blocking Google-Extended remove me from Google? No. Google says it doesn’t affect Google Search. It only covers Gemini training and Gemini’s use of your site in answers.
Should my startup block AI bots like the Times? Usually not the answer bots. Being mentioned in AI answers is how many people now find tools. Blocking training bots is a separate choice.
Run your own site through the free AI readiness checker, or see what AI actually says about your product with a DowseAI report. Questions: hello@dowseai.com.
Sources
- Google crawler docs (Google-Extended): https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
- Google, Search Console AI features control (3 Jun 2026, updated 31 Aug 2026): https://blog.google/products-and-platforms/products/search/new-opportunities-control-and-insights-for-website-owners/
- NYT and Amazon deal, Wirecutter excluded: https://techcrunch.com/2025/05/29/the-new-york-times-and-amazon-ink-ai-licensing-deal and https://www.axios.com/2025/05/30/nyt-amazon-ai-licensing-deal
- NYT sues Perplexity (5 Dec 2025): https://www.axios.com/2025/12/05/nyt-sues-perplexity-for-copyright-infringement