AI 公司使用爬虫收集网站内容,用于训练语言模型、生成搜索答案以及其他目的。域名根目录下的 robots.txt 文件会告诉这些爬虫它们应该或不应该访问哪些内容。当至少有一项策略设置为 Block(阻止)、Block on pages with ads(在含广告的页面上阻止),或对于 Training(训练)策略设置为 Disallow(禁止) 时,Bot Preference Sync 会根据您的 Search(搜索)、Agent 和 Training(训练)策略生成托管的 robots.txt 指令。
遵守 robots.txt 是自愿的。该文件表达了您的偏好,但它在技术层面上并不能阻止爬虫访问您的内容。一些爬虫运营商可能会忽略您的 robots.txt 指令(如 Disallow: / 等指令)并照样抓取您的内容。
Cloudflare 会检测您的源服务器是否已存在 robots.txt 文件,并进行相应调整 — 与您现有的文件合并,或从头开始创建一个。
如果你的网站已有 robots.txt 文件(通过 HTTP 200 响应验证),Cloudflare 会将托管的 robots.txt 添加到现有 robots.txt 之前,并将两者合并为单个响应。
例如,在不启用此功能的情况下,crawlstop.com 的 robots.txt 内容为:
User-agent: *
Disallow: /lp
Disallow: /feedback
Disallow: /langtest
Sitemap: https://www.crawlstop.com/sitemap.xml启用托管 robots.txt 后,Cloudflare 将在我们托管的内容之前添加我们的内容,生成类似于您在 https://www.crawlstop.com/robots.txt ↗ 上看到的内容。
# As a condition of accessing this website, you agree to abide by the
# following content signals:
# (a) If a content-signal = yes, you may collect content for the
# corresponding use.
# (b) If a content-signal = no, you may not collect content for the
# corresponding use.
# (c) If the website operator does not include a content signal for a
# corresponding use, the website operator neither grants nor restricts
# permission via content signal with respect to the corresponding use.
# The content signals and their meanings are:
# search: building a search index and providing search results (e.g., returning
# hyperlinks and short excerpts from your website's contents). Search
# does not include providing AI-generated search summaries.
# ai-input: inputting content into one or more AI models (e.g., retrieval
# augmented generation, grounding, or other real-time taking of
# content for generative AI search answers).
# ai-train: training or fine-tuning AI models.
# ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF
# RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT
# AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.
# BEGIN Cloudflare Managed content
User-Agent: *
Content-signal: search=yes, ai-train=no, use=reference
Allow: /
User-agent: Amazonbot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: meta-externalagent
Disallow: /
# END Cloudflare Managed Content
User-agent: *
Disallow: /lp
Disallow: /feedback
Disallow: /langtest
Sitemap: https://www.crawlstop.com/sitemap.xml如果您的网站没有 robots.txt 文件,Cloudflare 将创建一个新文件,其中包含针对已知 AI 爬虫的托管 Disallow 规则,并为您提供服务。
要为 AI 训练爬虫添加托管指令:
-
在 Cloudflare 仪表板中,前往 Security Settings(安全设置) 页面。
前往 Settings(设置) ↗ -
按 Bot traffic(Bot 流量) 过滤。
-
选择 Configure AI bot policies(配置 AI bot 策略)。
-
对于 Training(训练),选择 Disallow(禁止)。
-
开启 Enable Bot Preference Sync(启用 Bot Preference Sync)。
-
选择 Save(保存)。
启用 Bot Preference Sync 后,将 Training(训练) 设置为 Disallow(禁止) 会添加托管指令,并为 AI 训练爬虫启用边缘阻止。混合用途爬虫不在此边缘阻止范围内。
当 Search(搜索) 和 Agent 策略设置为 Block(阻止) 或 Block on pages with ads(在含广告的页面上阻止) 时,也会添加各自对应的托管指令。
内容信号 (Content Signals) 是 robots.txt 文件中的一组机器可读指令,用于分类爬虫如何使用您的内容。这三个类别是 search(构建搜索索引)、ai-input(将内容输入到 AI 模型中以获取实时答案)和 ai-train(训练或微调 AI 模型)。
对于没有自己的 robots.txt 文件且未使用托管 robots.txt 功能的免费版套餐域名,当爬虫请求您域名的 robots.txt 文件时,将显示内容信号策略 (Content Signals Policy)。
内容信号策略定义了这些类别,但并未表达关于您内容的任何特定偏好。要设置偏好(例如 ai-train=no),请配置您的 AI bot 策略并启用 Bot Preference Sync。
# As a condition of accessing this website, you agree to abide by the
# following content signals:
# (a) If a content-signal = yes, you may collect content for the
# corresponding use.
# (b) If a content-signal = no, you may not collect content for the
# corresponding use.
# (c) If the website operator does not include a content signal for a
# corresponding use, the website operator neither grants nor restricts
# permission via content signal with respect to the corresponding use.
# The content signals and their meanings are:
# search: building a search index and providing search results (e.g., returning
# hyperlinks and short excerpts from your website's contents). Search
# does not include providing AI-generated search summaries.
# ai-input: inputting content into one or more AI models (e.g., retrieval
# augmented generation, grounding, or other real-time taking of
# content for generative AI search answers).
# ai-train: training or fine-tuning AI models.
# ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF
# RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT
# AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.每当 Bot Preference Sync 向您的 robots.txt 文件添加托管指令时,Cloudflare 都会包含内容信号策略。
Cloudflare 正在测试 content-use,这是存在于您的 robots.txt 中的 Content Signals ↗ 的一个可选扩展。它在现有的 search、ai-input 和 ai-train 信号旁边添加了第四个字段,以描述爬虫在访问您的内容后可以保留和重用什么。该字段取以下三个值之一,从最不宽松到最宽松:
| 值 | 含义 |
|---|---|
use=immediate |
允许交互,但不存储和重用任何内容。 |
use=reference |
索引、摘录并链接回。 |
use=full |
总结并复制。 |
对于开启了托管 robots.txt 设置的客户,Cloudflare 会在托管内容中添加 use=reference,与现有的默认值 search=yes,ai-train=no 保持一致:
User-Agent: *
Content-signal: search=yes, ai-train=no, use=reference
Allow: /适用于 AI 爬虫的托管 robots.txt 适用于所有套餐方案。