{"id":6134,"date":"2011-09-01T06:00:16","date_gmt":"2011-08-31T21:00:16","guid":{"rendered":"http:\/\/www.suzukikenichi.com\/blog\/?p=6134"},"modified":"2011-09-01T05:16:29","modified_gmt":"2011-08-31T20:16:29","slug":"if-i-dont-need-to-block-crawlers-should-i-create-robots-txt-file","status":"publish","type":"post","link":"https:\/\/www.suzukikenichi.com\/blog\/if-i-dont-need-to-block-crawlers-should-i-create-robots-txt-file\/","title":{"rendered":"\u30af\u30ed\u30fc\u30e9\u3092\u30d6\u30ed\u30c3\u30af\u201c\u3057\u306a\u3044\u201d\u6642\u306erobots.txt\u306e\u8a2d\u5b9a"},"content":{"rendered":"<p class=\"p20-b al-r small\">[\u30ec\u30d9\u30eb\uff1a\u521d\u7d1a]<\/p>\n<blockquote><p>\u691c\u7d22\u30a8\u30f3\u30b8\u30f3\u306e\u30af\u30ed\u30fc\u30e9\u3092\u30d6\u30ed\u30c3\u30af\u3059\u308b\u5fc5\u8981\u304c\u306a\u3044\u6642\u306erobots.txt\u306e\u8a2d\u5b9a\u306f\u3001\u6b21\u306e3\u3064\u306e\u3046\u3061\u3069\u308c\u304c\u9069\u5207\u304b\uff1f<\/p>\n<ol>\n<li>\u4f55\u3082\u66f8\u3044\u3066\u3044\u306a\u3044robots.txt\u3092\u8a2d\u7f6e\u3059\u308b<\/li>\n<li>\u4e0b\u306e\u8a18\u8ff0\u3092\u66f8\u304f\n<pre style=\"overflow-x: auto\"><code class=\"green\">User-Agent: *\r\nDisallow:<\/code><\/pre>\n<\/li>\n<li>robots.txt\u305d\u306e\u3082\u306e\u3092\u7f6e\u304b\u306a\u3044<\/li>\n<\/ol>\n<\/blockquote>\n<p><a href=\"\/\/www.suzukikenichi.com\/blog\/how-to-create-robotstxt\/\">robots.txt<\/a>\u306b\u3064\u3044\u3066\u3001\u4e0a\u306e\u8cea\u554f\u306bGoogle\u306eMatt Cutts(\u30de\u30c3\u30c8\u30fb\u30ab\u30c3\u30c4)\u6c0f\u304c\u30d3\u30c7\u30aa\u3067\u56de\u7b54\u3057\u307e\u3057\u305f\u3002<\/p>\n<p><iframe loading=\"lazy\" width=\"560\" height=\"345\" src=\"\/\/www.youtube.com\/embed\/P7GY1fE5JQQ?rel=0\" frameborder=\"0\" allowfullscreen><\/iframe><\/p>\n<blockquote><p>1\u3064\u76ee\u304b2\u3064\u76ee\u306e\u3069\u3061\u3089\u304b\u304c\u9069\u5207\u3060\u308d\u3046\u30023\u3064\u76ee\u3060\u3068\u3001\u3072\u3087\u3063\u3068\u3057\u305f\u3089\u307b\u3093\u306e\u5c11\u3057\u3060\u3051\u3069\u5371\u967a\u304b\u3082\u3057\u308c\u306a\u3044\u3002<\/p>\n<p>robots.txt\u3092\u7f6e\u3044\u3066\u3044\u306a\u3044\u3068\u30b5\u30fc\u30d0\u30fc\u306e\u30db\u30b9\u30c6\u30a3\u30f3\u30b0\u4f1a\u793e\u304c404\u3092\u57cb\u3081\u308b\u305f\u3081\u306b\u4f55\u304b\u3057\u3066\u3057\u307e\u3044\u3001\u304a\u304b\u3057\u306a\u6319\u52d5\u3092\u8d77\u3053\u3059\u3053\u3068\u304c\u3042\u308b\u304b\u3089\u3060\u3002<br \/>\n\u307b\u3068\u3093\u3069\u306e\u5834\u5408\u50d5\u305f\u3061\u306f\u3046\u307e\u304f\u5bfe\u51e6\u3067\u304d\u308b\u3051\u3069\u30011%\u3068\u3044\u3048\u3069\u5371\u967a\u304c\u306a\u3044\u308f\u3051\u3058\u3083\u306a\u3044\u3002<\/p>\n<p>\u3060\u304b\u3089\u30d6\u30e9\u30f3\u30af\u306erobots.txt\u3092\u7f6e\u304f\u304b\u3001\u3059\u3079\u3066\u306e\u30a2\u30af\u30bb\u30b9\u3092\u30af\u30ed\u30fc\u30e9\u306b\u8a31\u53ef\u3057\u3066\u304a\u304f\u3068\u3044\u3044\u3002\u3069\u3061\u3089\u3067\u3082\u307e\u3063\u305f\u304f\u5909\u308f\u3089\u306a\u3044\u3002<\/p><\/blockquote>\n<p class=\"p40-t\">\u540c\u3058\u3088\u3046\u306a\u3053\u3068\u3092\u7e70\u308a\u8fd4\u3057\u3066\u3044\u308b\u306e\u3067\u8a33\u306f\u7aef\u6298\u3063\u3066\u307e\u3059\u304c\u3001\u30af\u30ed\u30fc\u30e9\u3092\u62d2\u5426\u3057\u306a\u3044\u306e\u3067\u3042\u308c\u3070<strong>\u7a7a\u3063\u307d\u306erobots.txt\u3092\u7f6e\u3044\u3066\u304a\u304f<\/strong>\u304b\u3001\u4f55\u3082\u62d2\u5426\u3057\u306a\u3044\u3001\u3064\u307e\u308a<strong>\u3059\u3079\u3066\u3092\u8a31\u53ef\u3059\u308b\u8a18\u8ff0\u3092\u66f8\u3044\u305frobots.txt\u3092\u7f6e\u3044\u3066\u304a\u304f<\/strong>\u3068\u3044\u3044\u3068\u3044\u3046\u30a2\u30c9\u30d0\u30a4\u30b9\u306b\u306a\u308a\u307e\u3059\u3002<\/p>\n<p>robots.txt\u3092\u7f6e\u304b\u306a\u304f\u3066\u3082\u3044\u3044\u3051\u308c\u3069\u3001\u4e07\u304c\u4e00\u306e\u5834\u5408\u304c\u3042\u308b\u306e\u3067\u304a\u3059\u3059\u3081\u306f\u3057\u306a\u3044\u3068\u306e\u3053\u3068\u3067\u3059\u3002<\/p>\n<p class=\"p40-t\">robots.txt\u3092\u7f6e\u304b\u306a\u3044\u6642\u306f\u3001robots.txt\u306e\u30ea\u30af\u30a8\u30b9\u30c8\u304c\u3042\u3063\u305f\u969b\uff08\u901a\u5e38\u30af\u30ed\u30fc\u30e9\u306f\u30a2\u30af\u30bb\u30b9\u3057\u305f\u3068\u304d\u306b\u3001\u6700\u521d\u306brobots.txt\u3092\u63a2\u3057\u307e\u3059\u3002\u30a2\u30af\u30bb\u30b9\u304c\u7981\u6b62\u3055\u308c\u3066\u3044\u308b\u30b3\u30f3\u30c6\u30f3\u30c4\u304c\u306a\u3044\u304b\u78ba\u8a8d\u3059\u308b\u305f\u3081\u3067\u3059\uff09\u3001404\u30a8\u30e9\u30fc\u307e\u305f\u306f410\u30a8\u30e9\u30fc\u3092\u8fd4\u3059\u3088\u3046\u306b\u3057\u307e\u3059\u3002<\/p>\n<p>404\/410\u3092\u78ba\u5b9f\u306b\u8fd4\u305b\u3070\u554f\u984c\u306f\u767a\u751f\u3057\u307e\u305b\u3093\u3002<\/p>\n<p>\u3067\u3082\u4f8b\u3048\u3070\u3001500\u756a\u53f0\u306e\u30a8\u30e9\u30fc\u3092\u8fd4\u3057\u305f\u308a\u3059\u308b\u3068\u30e9\u30f3\u30ad\u30f3\u30b0\u304c\u4e0b\u304c\u3063\u305f\u308a\u3001\u3072\u3069\u3044\u3068\u304d\u306f\u30a4\u30f3\u30c7\u30c3\u30af\u30b9\u304b\u3089\u6d88\u3048\u3066\u3057\u307e\u3046\u3053\u3068\u304c\u3042\u308a\u307e\u3059\u3002<\/p>\n<p>\u3053\u306e\u3042\u305f\u308a\u306f\u3001Web\u62c5\u5f53\u8005Forum\u306e\u9023\u8f09\u3067\u66f8\u3044\u305f<a href=\"http:\/\/web-tan.forum.impressrd.jp\/e\/2011\/07\/29\/10783\/page%3D0,1#f4\">\u3053\u306e\u8a18\u4e8b<\/a>\u3068<a href=\"http:\/\/web-tan.forum.impressrd.jp\/e\/2011\/08\/19\/10893\/page%3D0%2C1#f2\">\u3053\u306e\u8a18\u4e8b<\/a>\u3092\u53c2\u7167\u3057\u3066\u304f\u3060\u3055\u3044\u3002<\/p>\n<p>\u50d5\u304c\u3053\u308c\u307e\u3067\u53d7\u3051\u305f\u554f\u3044\u5408\u308f\u305b\u3067\u3001robots.txt\u304c302\uff08\u304b301\uff09\u3067\u30c8\u30c3\u30d7\u30da\u30fc\u30b8\u306b\u30ea\u30c0\u30a4\u30ec\u30af\u30c8\u3055\u308c\u3066\u3044\u305f\u308a\u3001\u30b5\u30fc\u30d0\u30fc\u306e\u30d5\u30a1\u30a4\u30a2\u30fc\u30a6\u30a9\u30fc\u30eb\u306e\u8a2d\u5b9a\u304b\u4f55\u304b\u3067Googlebot\u3060\u3051\u304crobots.txt\u306b\u30a2\u30af\u30bb\u30b9\u3067\u304d\u306a\u304b\u3063\u305f\u308a\u3057\u305f\u305f\u3081\u306b\u3001\u30a4\u30f3\u30c7\u30c3\u30af\u30b9\u304c\u6d88\u3048\u305f\u30c8\u30e9\u30d6\u30eb\u304c\u3042\u308a\u307e\u3057\u305f\u3002<\/p>\n<p>\u81ea\u5206\u306e\u30b5\u30a4\u30c8\u306erobots.txt\u306b\u30a2\u30af\u30bb\u30b9\u3057\u3066\u307f\u3066\u3001\u8a2d\u7f6e\u3057\u3066\u3044\u308b\u5834\u5408\u306f200\u756a\u306eHTTP\u30b9\u30c6\u30fc\u30bf\u30b9\u30b3\u30fc\u30c9\u304c\u6b63\u3057\u304f\u8fd4\u3055\u308c\u3066\u3044\u308b\u304b\u3001\u8a2d\u7f6e\u3057\u3066\u3044\u306a\u3044\u5834\u5408\u306f404\/410\u30a8\u30e9\u30fc\u304c\u8fd4\u3055\u308c\u3066\u3044\u308b\u304b\u3092\u78ba\u8a8d\u3057\u3066\u304a\u304d\u307e\u3057\u3087\u3046\u3002<\/p>\n<p>Google\u30a6\u30a7\u30d6\u30de\u30b9\u30bf\u30fc\u30c4\u30fc\u30eb\u306e\u300c\u30af\u30ed\u30fc\u30e9\u306e\u30a2\u30af\u30bb\u30b9\u300d\u3067\u72b6\u614b\u3092\u78ba\u8a8d\u3067\u304d\u307e\u3059\u306d\u3002<br \/>\n200\u304c\u8fd4\u3055\u308c\u3066\u3044\u308b\u6642\u306f\u3001\u8a18\u8ff0\u304c\u6b63\u3057\u304f\u8aad\u307f\u53d6\u3089\u308c\u3066\u3044\u308b\u304b\u3082\u4f75\u305b\u3066\u78ba\u8a8d\u3057\u3066\u304a\u304f\u3068\u3044\u3044\u3067\u3057\u3087\u3046\u3002<\/p>\n<p><a href=\"\/\/www.suzukikenichi.com\/images\/crawler-access-to-robots-txt-in-webmastertools.png\"><img decoding=\"async\" src=\"\/\/www.suzukikenichi.com\/images\/crawler-access-to-robots-txt-in-webmastertools2.png\" alt=\"Google\u30a6\u30a7\u30d6\u30de\u30b9\u30bf\u30fc\u30c4\u30fc\u30eb\u306e\u300c\u30af\u30ed\u30fc\u30e9\u306e\u30a2\u30af\u30bb\u30b9\u300d\" \/><\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>\u30af\u30ed\u30fc\u30e9\u306e\u30a2\u30af\u30bb\u30b9\u3092\u4f55\u3082\u62d2\u5426\u3057\u306a\u3044\u306e\u3067robots.txt\u304c\u5fc5\u8981\u306a\u3044\u30b5\u30a4\u30c8\u3067\u306frobots.txt\u3092\u3069\u3046\u3059\u308b\u3068\u3044\u3044\u306e\u3060\u308d\u3046\u304b\uff1f1.\u4f55\u3082\u8a18\u8ff0\u3057\u306a\u3044robots.txt\u3092\u8a2d\u7f6e\u3059\u308b\u30012.\u300cUser-Agent: *, Disallow:\u300d\u3092\u8a18\u8ff0\u3059\u308b\u30013.robots.txt\u3092\u7f6e\u304b\u306a\u3044\u3002\u3069\u308c\u304c\u9069\u5207\u304b\uff1f<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[62],"class_list":["post-6134","post","type-post","status-publish","format-standard","category-google-seo","tag-robotstxt"],"_links":{"self":[{"href":"https:\/\/www.suzukikenichi.com\/blog\/wp-json\/wp\/v2\/posts\/6134","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.suzukikenichi.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.suzukikenichi.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.suzukikenichi.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.suzukikenichi.com\/blog\/wp-json\/wp\/v2\/comments?post=6134"}],"version-history":[{"count":0,"href":"https:\/\/www.suzukikenichi.com\/blog\/wp-json\/wp\/v2\/posts\/6134\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.suzukikenichi.com\/blog\/wp-json\/wp\/v2\/media?parent=6134"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.suzukikenichi.com\/blog\/wp-json\/wp\/v2\/categories?post=6134"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.suzukikenichi.com\/blog\/wp-json\/wp\/v2\/tags?post=6134"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}