{"id":13139,"date":"2023-04-03T07:10:49","date_gmt":"2023-04-02T22:10:49","guid":{"rendered":"https:\/\/www.suzukikenichi.com\/blog\/?p=13139"},"modified":"2023-05-25T06:27:13","modified_gmt":"2023-05-24T21:27:13","slug":"how-to-block-chatgpt-and-common-crawl-by-robots-txt","status":"publish","type":"post","link":"https:\/\/www.suzukikenichi.com\/blog\/how-to-block-chatgpt-and-common-crawl-by-robots-txt\/","title":{"rendered":"ChatGPT\u3068Common Crawl\u306e\u30b5\u30a4\u30c8\u3078\u306e\u30a2\u30af\u30bb\u30b9\u3092\u62d2\u5426\u3059\u308b\u65b9\u6cd5"},"content":{"rendered":"<p class=\"p20-b al-r small\">[\u30ec\u30d9\u30eb: \u4e0a\u7d1a]<\/p>\n<p class=\"p60-b\">\u3053\u306e\u8a18\u4e8b\u3067\u306f\u3001ChatGPT \u3068 Common Crawl \u304c\u30b5\u30a4\u30c8\u306b\u30a2\u30af\u30bb\u30b9\u3059\u308b\u306e\u3092\u9632\u3050\u65b9\u6cd5\u3092\u89e3\u8aac\u3057\u307e\u3059\u3002<\/p>\n<h2>ChatGPT \u30d7\u30e9\u30b0\u30a4\u30f3\u3092 robots.txt \u3067\u30d6\u30ed\u30c3\u30af<\/h2>\n<p>ChatGPT \u81ea\u4f53\u304c\u30b5\u30a4\u30c8\u3092\u30af\u30ed\u30fc\u30eb\u3059\u308b\u3053\u3068\u306f\u3042\u308a\u307e\u305b\u3093\u3002<br \/>\u3057\u304b\u3057\u3001\u30d7\u30e9\u30b0\u30a4\u30f3\u304c\u30b5\u30a4\u30c8\u306b\u30a2\u30af\u30bb\u30b9\u3059\u308b\u3053\u3068\u304c\u3042\u308a\u307e\u3059\u3002<br \/>ChatGPT \u30d7\u30e9\u30b0\u30a4\u30f3\u306e\u30a2\u30af\u30bb\u30b9\u306f robots.txt \u3067\u30d6\u30ed\u30c3\u30af\u3067\u304d\u307e\u3059\u3002<\/p>\n<p>UA\uff08\u30e6\u30fc\u30b6\u30fc\u30a8\u30fc\u30b8\u30a7\u30f3\u30c8\uff09\u306f\u3001<code>ChatGPT-User<\/code> \u3067\u3059\u3002<\/p>\n<p>\u30b5\u30a4\u30c8\u3078\u306e\u30a2\u30af\u30bb\u30b9\u3092\u5b8c\u5168\u306b\u62d2\u5426\u3059\u308b\u5834\u5408\u306f\u3001\u6b21\u306e\u3088\u3046\u306b robots.txt \u306b\u8a18\u8ff0\u3057\u307e\u3059\u3002<\/p>\n<pre style=\"overflow-x: auto\"><code class=\"green\">User-agent: ChatGPT-User\r\nDisallow: \/<\/code><\/pre>\n<p>\u4e00\u90e8\u306e URL \u306b\u30a2\u30af\u30bb\u30b9\u3055\u305b\u305f\u304f\u306a\u3051\u308c\u3070\u3001<a href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/robots\/create-robots-txt\">robots.txt \u306e\u8a18\u8ff0\u30eb\u30fc\u30eb<\/a>\u306b\u5f93\u3063\u3066\u8a18\u8ff0\u3057\u307e\u3059\u3002<\/p>\n<pre style=\"overflow-x: auto\"><code class=\"green\">User-agent: ChatGPT-User\r\nDisallow: \/no-chatgpt\/<\/code><\/pre>\n<div class=\"small\">&#x1f4dd;<code>no-chatgpt<\/code> \u30c7\u30a3\u30ec\u30af\u30c8\u30ea\u914d\u4e0b\u306e\u30da\u30fc\u30b8\u3078\u306e\u30a2\u30af\u30bb\u30b9\u3092\u62d2\u5426<\/div>\n<p>robots.txt \u306e\u62e1\u5f35\u4ed5\u69d8\u306e <code>Allow<\/code> \u30c7\u30a3\u30ec\u30af\u30c6\u30a3\u30d6\u3092 ChatGPT-User \u306f\u30b5\u30dd\u30fc\u30c8\u3057\u307e\u3059\u3002<br \/>\u30b5\u30a4\u30c8\u5168\u4f53\u3078\u306e\u30a2\u30af\u30bb\u30b9\u306f\u57fa\u672c\u7684\u306b\u62d2\u5426\u3059\u308b\u3082\u306e\u306e\u3001\u7279\u5b9a\u306e URL \u3078\u306e\u30a2\u30af\u30bb\u30b9\u3060\u3051\u306f\u8a31\u53ef\u3059\u308b\u69cb\u6210\u3082\u53ef\u80fd\u3067\u3059\u3002<br \/><code>Allow<\/code> \u30c7\u30a3\u30ec\u30af\u30c6\u30a3\u30d6\u3092\u4f7f\u3044\u307e\u3059\u3002<\/p>\n<pre style=\"overflow-x: auto\"><code class=\"green\">User-agent: ChatGPT-User\r\nDisallow: \/\r\nAllow: \/directory-1\/\r\nAllow: \/directory-2\/<\/code><\/pre>\n<div class=\"small\">&#x1f4dd;<code>directory-1<\/code> \u3068 <code>directory-2<\/code> \u306e\u30c7\u30a3\u30ec\u30af\u30c8\u30ea\u914d\u4e0b\u306e URL \u306b\u3060\u3051\u30a2\u30af\u30bb\u30b9\u3092\u8a31\u53ef<\/div>\n<p>\u81ea\u5206\u306e\u30b5\u30a4\u30c8\u306e\u60c5\u5831\u3092 ChatGPT \u306b\u4f7f\u308f\u305b\u305f\u304f\u306a\u3044\u3068\u3044\u3046\u65b9\u306f robots.txt \u306e\u5229\u7528\u3092\u691c\u8a0e\u3057\u3066\u304f\u3060\u3055\u3044\u3002<\/p>\n<p>\u307e\u305f\u3001OpenAI \u306e IP \u30a2\u30c9\u30ec\u30b9\u3082\u516c\u958b\u3055\u308c\u3066\u3044\u307e\u3059\u3002<br \/>IP \u30a2\u30c9\u30ec\u30b9\u3067\u306e\u5236\u5fa1\u3082\u3067\u304d\u307e\u3059\u3002<\/p>\n<p class=\"p40-b\">ChatGPT \u306e\u30e6\u30fc\u30b6\u30fc\u30a8\u30fc\u30b8\u30a7\u30f3\u30c8\u304a\u3088\u3073 IP \u30ec\u30f3\u30b8\u306b\u3064\u3044\u3066\u306f <a href=\"https:\/\/platform.openai.com\/docs\/plugins\/bot\">OpenAI \u306e\u30c9\u30ad\u30e5\u30e1\u30f3\u30c8<\/a>\u3067\u78ba\u8a8d\u3067\u304d\u307e\u3059\u3002<\/p>\n<h2 id=\"common-crawl\">Common Crawl \u306e\u30af\u30ed\u30fc\u30e9\u3092 robots.txt \u3067\u30d6\u30ed\u30c3\u30af<\/h2>\n<p><a href=\"https:\/\/commoncrawl.org\/\">Common Crawl<\/a>\uff08\u30b3\u30e2\u30f3\u30fb\u30af\u30ed\u30fc\u30eb\uff09\u306f\u3001\u30a6\u30a7\u30d6\u3092\u30af\u30ed\u30fc\u30eb\u3057\u305d\u306e\u30c7\u30fc\u30bf\u306e\u30a2\u30fc\u30ab\u30a4\u30d6\u3092\u63d0\u4f9b\u3059\u308b\u975e\u55b6\u5229\u7d44\u7e54\u3067\u3059\u3002<br \/>\u30a6\u30a7\u30d6\u306e\u81a8\u5927\u306a\u91cf\u306e\u30c7\u30fc\u30bf\u3092\u84c4\u7a4d\u3057\u3066\u304a\u308a\u3001\u591a\u304f\u306e LLM \u304c Common Crawl \u306e\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u3092 AI \u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u306b\u5229\u7528\u3057\u3066\u3044\u307e\u3059\u3002<br \/>Google \u3084 Microsoft \u3082\u5229\u7528\u3057\u3066\u3044\u307e\u3059\u3002<\/p>\n<p>\u3057\u304b\u3057\u3001\u81ea\u5206\u305f\u3061\u304c\u516c\u958b\u3057\u305f\u30b3\u30f3\u30c6\u30f3\u30c4\u3092\u7121\u8a31\u53ef\u3067\u4f7f\u308f\u308c\u3066\u3044\u308b\u3068\u3057\u3066\u3001<a href=\"https:\/\/www.bloomberg.com\/news\/articles\/2023-02-17\/openai-is-faulted-by-media-for-using-articles-to-train-chatgpt?leadSource=uverify%20wall#xj4y7vzkg\">Wall Street Journal \u3084 CNN \u306e\u3088\u3046\u306b\u61f8\u5ff5\u3092\u62b1\u3044\u3066<\/a>\u3044\u308b\u30d1\u30d6\u30ea\u30c3\u30b7\u30e3\u30fc\u3082\u5b58\u5728\u3057\u307e\u3059\u3002<\/p>\n<p>ChatGPT \u3068\u540c\u3058\u3088\u3046\u306b <a href=\"https:\/\/commoncrawl.org\/big-picture\/frequently-asked-questions\/#:~:text=How%20can%20I%20block%20this%20bot%3F\">robots.txt \u3067\u3001Common Crawl \u304c\u81ea\u5206\u306e\u30b5\u30a4\u30c8\u306b\u30a2\u30af\u30bb\u30b9\u3059\u308b\u3053\u3068\u3092\u62d2\u5426<\/a>\u3067\u304d\u307e\u3059\u3002<\/p>\n<p>Common Crawl \u306e UA \u306f <code>CCBot<\/code> \u3067\u3059\u3002<\/p>\n<p>\u6b21\u306e\u3088\u3046\u306b robots.txt \u306b\u8a18\u8ff0\u3059\u308b\u3068 CCBot \u306e\u30af\u30ed\u30fc\u30eb\u3092\u5b8c\u5168\u306b\u30d6\u30ed\u30c3\u30af\u3067\u304d\u307e\u3059\u3002<\/p>\n<pre style=\"overflow-x: auto\"><code class=\"green\">User-agent: CCBot\r\nDisallow: \/<\/code><\/pre>\n<p>\u81ea\u5206\u306e\u30b5\u30a4\u30c8\u3067\u516c\u958b\u3057\u3066\u3044\u308b\u30b3\u30f3\u30c6\u30f3\u30c4\u3092\u7121\u8a31\u53ef\u3067 Common Crawl \u306b\u53d6\u5f97\u3055\u308c\u305f\u304f\u306a\u3044\u3068\u3044\u3046\u306e\u3067\u3042\u308c\u3070\u3001robots.txt \u306e\u4f7f\u7528\u3092\u691c\u8a0e\u3059\u308b\u3068\u3044\u3044\u3067\u3057\u3087\u3046\u3002<\/p>\n","protected":false},"excerpt":{"rendered":"<p>ChatGPT \u3068 Common Crawl \u304c\u30b5\u30a4\u30c8\u306b\u30a2\u30af\u30bb\u30b9\u3059\u308b\u306e\u3092\u9632\u3050\u65b9\u6cd5\u3092\u89e3\u8aac\u3059\u308b\u3002<\/p>\n","protected":false},"author":1,"featured_media":13140,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1728,8],"tags":[62,1708,1719],"class_list":["post-13139","post","type-post","status-publish","format-standard","has-post-thumbnail","category-ai-chat","category-net-marketing","tag-robotstxt","tag-chatgpt","tag-common-crawl"],"_links":{"self":[{"href":"https:\/\/www.suzukikenichi.com\/blog\/wp-json\/wp\/v2\/posts\/13139","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.suzukikenichi.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.suzukikenichi.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.suzukikenichi.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.suzukikenichi.com\/blog\/wp-json\/wp\/v2\/comments?post=13139"}],"version-history":[{"count":0,"href":"https:\/\/www.suzukikenichi.com\/blog\/wp-json\/wp\/v2\/posts\/13139\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.suzukikenichi.com\/blog\/wp-json\/wp\/v2\/media\/13140"}],"wp:attachment":[{"href":"https:\/\/www.suzukikenichi.com\/blog\/wp-json\/wp\/v2\/media?parent=13139"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.suzukikenichi.com\/blog\/wp-json\/wp\/v2\/categories?post=13139"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.suzukikenichi.com\/blog\/wp-json\/wp\/v2\/tags?post=13139"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}