{"id":15118,"date":"2025-05-30T07:10:17","date_gmt":"2025-05-29T22:10:17","guid":{"rendered":"https:\/\/www.suzukikenichi.com\/blog\/?p=15118"},"modified":"2025-05-30T07:08:10","modified_gmt":"2025-05-29T22:08:10","slug":"a-look-at-how-web-crawlers-have-evolved","status":"publish","type":"post","link":"https:\/\/www.suzukikenichi.com\/blog\/a-look-at-how-web-crawlers-have-evolved\/","title":{"rendered":"\u30a6\u30a7\u30d6\u30af\u30ed\u30fc\u30e9\u30fc\u306f\u3069\u306e\u3088\u3046\u306b\u9032\u5316\u3057\u3066\u304d\u305f\u306e\u304b\uff1f Google\u793e\u54e1\u304c\u8a9e\u308b\u305d\u306e\u6b74\u53f2"},"content":{"rendered":"<p class=\"p20-b al-r small\">[\u30ec\u30d9\u30eb: \u4e2d\u7d1a]<\/p>\n<p>Google \u691c\u7d22\u30ea\u30ec\u30fc\u30b7\u30e7\u30f3\u30ba\u30c1\u30fc\u30e0\u306b\u3088\u308b\u30dd\u30c3\u30c9\u30ad\u30e3\u30b9\u30c8\u30b7\u30ea\u30fc\u30ba\u306e\u300eSearch Off the Record\u300f\u306e<a href=\"https:\/\/search-off-the-record.libsyn.com\/how-googlebot-crawls-the-web\">\u30a8\u30d4\u30bd\u30fc\u30c9 92<\/a> \u3067\u306f\u3001Gary Illyes\uff08\u30b2\u30a4\u30ea\u30fc\u30fb\u30a4\u30ea\u30fc\u30b9\uff09\u6c0f\u3068 Matin Splitt\uff08\u30de\u30fc\u30c6\u30a3\u30f3\u30fb\u30b9\u30d7\u30ea\u30c3\u30c8\uff09\u6c0f\u304c\u3001\u30a6\u30a7\u30d6\u30af\u30ed\u30fc\u30e9\u30fc\u306e\u6b74\u53f2\u3001\u8a2d\u8a08\u3001\u305d\u3057\u3066\u672a\u6765\u306b\u3064\u3044\u3066\u30c7\u30a3\u30b9\u30ab\u30c3\u30b7\u30e7\u30f3\u3057\u307e\u3057\u305f\u3002<\/p>\n<p class=\"p60-b\">Google \u306e\u30af\u30ed\u30fc\u30e9\u30fc\u306e\u9032\u5316\u3092\u305f\u3069\u308a\u3001\u5e2f\u57df\u5e45\u3068\u30db\u30b9\u30c8\u8ca0\u8377\u306b\u95a2\u3059\u308b\u8003\u616e\u4e8b\u9805\u3092\u5f37\u8abf\u3057\u3001\u30a6\u30a7\u30d6\u306e\u6df7\u96d1\u304c\u5897\u3059\u4e2d\u3067\u73fe\u308c\u3066\u3044\u308b\u65b0\u305f\u306a\u8ab2\u984c\u306b\u3064\u3044\u3066 2 \u4eba\u306f\u63a2\u3063\u3066\u3044\u307e\u3059\u3002<\/p>\n<h2>\u521d\u671f\u306e\u30a6\u30a7\u30d6\u30af\u30ed\u30fc\u30ea\u30f3\u30b0<\/h2>\n<ul class=\"p40-b\">\n<li>1994 \u5e74\u306e\u5b66\u8853\u30af\u30ed\u30fc\u30e9\u300cWorld-Wide Web Worm\uff08WWWW\uff09\u300d\u306f\u7d04 11 \u4e07\u30da\u30fc\u30b8\u3092\u3001\u305d\u306e\u3042\u3068\u306b\u7d9a\u3044\u305f WebCrawler \u306f 200 \u4e07\u30da\u30fc\u30b8\u3092\u30a4\u30f3\u30c7\u30c3\u30af\u30b9\u5316\u3057\u305f\u3002\u3044\u305a\u308c\u3082\u4eca\u65e5\u306e\u57fa\u6e96\u304b\u3089\u3059\u308b\u3068\u3054\u304f\u308f\u305a\u304b\u306a\u6570\u3060\u304c\u3001\u5f53\u6642\u3068\u3057\u3066\u306f\u5de8\u5927\u3060\u3063\u305f\u3002<\/li>\n<li>Google \u306e\u524d\u8eab BackRub\uff081996\uff09\u306f\u30b7\u30a7\u30eb\u30b9\u30af\u30ea\u30d7\u30c8\u98a8\u306e\u5358\u7d14\u306a\u30d5\u30a7\u30c3\u30c1\u30e3\u3092\u7528\u3044\u3001\u305d\u306e\u5f8c\u5c02\u7528\u30af\u30ed\u30fc\u30e9\u30fc\u300cGooglebot\u300d\u304c\u8a95\u751f\u3002<\/li>\n<li>\u5f53\u6642\u306e\u30da\u30fc\u30b8\u30b5\u30a4\u30ba\u306f\u5e73\u5747 \u7d04 7 kB \u3067\u3001\u6570\u5341\u4e07\u30da\u30fc\u30b8\u3092\u53d6\u5f97\u3057\u3066\u3082\u5e2f\u57df\u8ca0\u62c5\u306f\u5c0f\u3055\u304b\u3063\u305f\u3002<\/li>\n<\/ul>\n<h2>\u5358\u4e00\u30af\u30ed\u30fc\u30e9\u304b\u3089\u5171\u6709\u30a4\u30f3\u30d5\u30e9\u3078<\/h2>\n<ul class=\"p40-b\">\n<li>\u521d\u671f\u306f Google \u306e\u5168\u30d7\u30ed\u30c0\u30af\u30c8\u304c Googlebot \u3092\u4f7f\u3044\u56de\u3057\u3001\u30b5\u30a4\u30c8\u7ba1\u7406\u8005\u306f\u76ee\u7684\u306e\u5224\u5225\u304c\u56f0\u96e3\u3060\u3063\u305f\u3002<\/li>\n<li>2006 \u5e74\u3054\u308d\u306b\u88fd\u54c1\u5225\u30e6\u30fc\u30b6\u30fc\u30a8\u30fc\u30b8\u30a7\u30f3\u30c8\uff08\u4f8b\uff1aAdsBot\uff09\u3092\u5c0e\u5165\u3057\u3064\u3064\u3001\u30ed\u30dc\u30c3\u30c8\u6392\u9664\u30eb\u30fc\u30eb (robots.txt) \u3068\u30db\u30b9\u30c8\u8ca0\u8377\u5236\u5fa1\u3092\u7d71\u4e00\u3057\u305f\u30d0\u30c3\u30af\u30a8\u30f3\u30c9\u57fa\u76e4\u306b\u7d71\u5408\u3002<\/li>\n<li>\u3053\u306e\u5171\u901a\u30d7\u30e9\u30c3\u30c8\u30d5\u30a9\u30fc\u30e0\u306b\u3088\u308a\u3001\u30a8\u30f3\u30b8\u30cb\u30a2\u306f\u30ab\u30b9\u30bf\u30e0\u30e6\u30fc\u30b6\u30fc\u30a8\u30fc\u30b8\u30a7\u30f3\u30c8\u6587\u5b57\u5217\u3092\u6307\u5b9a\u3057\u3064\u3064\u3001\u5171\u901a\u306e\u793c\u5100\u6b63\u3057\u3055\u306e\u30eb\u30fc\u30eb\u3092\u7d99\u627f\u3067\u304d\u308b\u305f\u3081\u3001\u30e1\u30f3\u30c6\u30ca\u30f3\u30b9\u306e\u8907\u96d1\u3055\u304c\u8efd\u6e1b\u3055\u308c\u3001\u300c\u4e0d\u6b63\u306a\u300d\u72ec\u81ea\u958b\u767a\u30af\u30ed\u30fc\u30e9\u30fc\u306e\u767a\u751f\u3092\u9632\u3052\u308b\u3002<\/li>\n<\/ul>\n<h2>\u793c\u5100\u6b63\u3057\u3055\u3001\u5e2f\u57df\u5e45\u3001\u30af\u30ed\u30fc\u30eb\u30d0\u30b8\u30a7\u30c3\u30c8<\/h2>\n<ul class=\"p40-b\">\n<li>\u8cac\u4efb\u3042\u308b\u30af\u30ed\u30fc\u30e9\u30fc\u306f Robots Exclusion Protocol\uff08\u30ed\u30dc\u30c3\u30c8\u6392\u9664\u30d7\u30ed\u30c8\u30b3\u30eb\uff09\u3092\u5c0a\u91cd\u3057\u3001\u30db\u30b9\u30c8\u304c\u8ca0\u8377\u3092\u793a\u3057\u305f\u5834\u5408\u306b\u306f\u52d5\u7684\u306b\u30af\u30ed\u30fc\u30eb\u901f\u5ea6\u3092\u8abf\u6574\u3059\u308b\u3053\u3068\u3067\u3001\u5076\u767a\u7684\u306a DoS\uff08\u30b5\u30fc\u30d3\u30b9\u62d2\u5426\uff09\u653b\u6483\u3092\u56de\u907f\u3059\u308b\u3002<\/li>\n<li>\u30b5\u30a4\u30c8\u306e\u30af\u30ed\u30fc\u30eb\u30d0\u30b8\u30a7\u30c3\u30c8\u306f\u3001\u305d\u306e\u30b5\u30a4\u30ba\u3068\u69cb\u6210\u306b\u4f9d\u5b58\u3059\u308b\u3002\u5358\u4e00\u30b5\u30a4\u30c8\u306e\u5834\u5408\u3001URL \u6570\u304c\u7d04 100 \u4e07\u672a\u6e80\u3067\u3042\u308c\u3070\u591a\u304f\u306e\u5834\u5408\u5feb\u9069\u306a\u72b6\u614b\u3092\u4fdd\u3066\u308b\u3002<\/li>\n<li>Google \u306f\u6700\u8fd1\u3001\u30ea\u30af\u30a8\u30b9\u30c8\u3054\u3068\u306e\u30aa\u30fc\u30d0\u30fc\u30d8\u30c3\u30c9\u3092 7 \u30d0\u30a4\u30c8\u524a\u6e1b\u3057\u305f\u304c\u3001\u65b0\u3057\u3044 AI \u6a5f\u80fd\u306b\u3088\u3063\u3066\u6642\u306b\u306f 8 \u30d0\u30a4\u30c8\u304c\u8ffd\u52a0\u3055\u308c\u308b\u3053\u3068\u3082\u3042\u308a\u3001\u7d76\u3048\u9593\u306a\u3044\u6700\u9069\u5316\u304c\u884c\u308f\u308c\u3066\u3044\u308b\u3002<\/li>\n<\/ul>\n<h2>\u30e6\u30fc\u30b6\u30fc\u4e3b\u5c0e\u30d5\u30a7\u30c3\u30c1 vs. \u81ea\u52d5\u30af\u30ed\u30fc\u30e9<\/h2>\n<ul class=\"p40-b\">\n<li>Search Console \u306e\u78ba\u8a8d\u306e\u3088\u3046\u306a\u30bf\u30b9\u30af\u306f\u3001\u30e6\u30fc\u30b6\u30fc\u306b\u4ee3\u308f\u3063\u3066\u5373\u5ea7\u306b\u53d6\u5f97\u51e6\u7406\u3092\u884c\u3044\u3001\u591a\u304f\u306e\u5834\u5408 robots.txt \u3084\u30ad\u30e5\u30fc\u306e\u9045\u5ef6\u3092\u8fc2\u56de\u3059\u308b\u3002<\/li>\n<li>\u5bfe\u7167\u7684\u306b\u3001\u30e9\u30a4\u30d6\u30c6\u30b9\u30c8\u306f\u3001\u3088\u308a\u9ad8\u3044\u512a\u5148\u5ea6\u3067\u306f\u3042\u308b\u3082\u306e\u306e\u3001\u4f9d\u7136\u3068\u3057\u3066\u901a\u5e38\u306e\u30af\u30ed\u30fc\u30e9\u30fc\u30ad\u30e5\u30fc\u3092\u4f7f\u7528\u3059\u308b\u3002<\/li>\n<li>\u3053\u308c\u3089\u306e\u30e2\u30fc\u30c9\u3092\u533a\u5225\u3059\u308b\u3053\u3068\u3067\u3001\u30ec\u30a4\u30c6\u30f3\u30b7\uff08\u9045\u5ef6\uff09\u8981\u4ef6\u3068\u30b5\u30a4\u30c8\u7ba1\u7406\u8005\u304b\u3089\u306e\u6307\u793a\u3068\u306e\u5c0a\u91cd\u3068\u306e\u30d0\u30e9\u30f3\u30b9\u3092\u53d6\u3063\u3066\u3044\u308b\u3002<\/li>\n<\/ul>\n<h2>\u4eca\u5f8c\u306e\u5c55\u671b<\/h2>\n<ul class=\"p40-b\">\n<li>\u30af\u30ed\u30fc\u30e9\u30fc\u3084 AI \u30a8\u30fc\u30b8\u30a7\u30f3\u30c8\u3001\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u53ce\u96c6\u30d7\u30ed\u30b0\u30e9\u30e0\u3092\u8ab0\u3082\u304c\u7acb\u3061\u4e0a\u3052\u3089\u308c\u308b\u3088\u3046\u306b\u306a\u308a\u3001\u30a6\u30a7\u30d6\u306f\u5897\u5927\u3059\u308b\u6df7\u96d1\u306b\u76f4\u9762\u3057\u3066\u3044\u308b\u3002<\/li>\n<li>\u73fe\u5728\u306e HTTP\/2\u3001\u5c06\u6765\u306e HTTP\/3 \u306e\u3088\u3046\u306a\u30d7\u30ed\u30c8\u30b3\u30eb\u306e\u9032\u6b69\u306f\u591a\u91cd\u5316\u3092\u6539\u5584\u3059\u308b\u304c\u3001\u30a4\u30f3\u30c7\u30c3\u30af\u30b9\u4f5c\u6210\u3068\u914d\u4fe1\u3068\u3044\u3046\u4e2d\u6838\u7684\u306a\u51e6\u7406\u30b3\u30b9\u30c8\u306e\u554f\u984c\u3092\u89e3\u6c7a\u3059\u308b\u3082\u306e\u3067\u306f\u306a\u3044\u3002<\/li>\n<li>Common Crawl \u306e\u3088\u3046\u306a\u30b3\u30df\u30e5\u30cb\u30c6\u30a3\u30d7\u30ed\u30b8\u30a7\u30af\u30c8\u306f\u5171\u6709\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u3092\u63d0\u4f9b\u3057\u3066\u3044\u308b\u3002\u73fe\u72b6\u306e\u3088\u3046\u306b\u3001\u591a\u304f\u306e\u5229\u7528\u8005\u304c\u5b8c\u5168\u306b\u72ec\u7acb\u3057\u305f\u30af\u30ed\u30fc\u30eb\u3092\u5b9f\u884c\u3059\u308b\u4ee3\u308f\u308a\u306b\u3001\u3053\u3046\u3057\u305f\u4e2d\u592e\u96c6\u6a29\u5316\u3055\u308c\u305f\u30a2\u30fc\u30ab\u30a4\u30d6\u306b\u4f9d\u5b58\u3059\u308b\u3088\u3046\u306b\u306a\u308c\u3070\u3001\u91cd\u8907\u30c8\u30e9\u30d5\u30a3\u30c3\u30af\u3092\u524a\u6e1b\u3067\u304d\u308b\u53ef\u80fd\u6027\u304c\u3042\u308b\u3002<\/li>\n<\/ul>\n<h2>\u4e3b\u8981\u30dd\u30a4\u30f3\u30c8\u307e\u3068\u3081<\/h2>\n<ul>\n<li><strong>\u30b9\u30b1\u30fc\u30eb\u306e\u62e1\u5927<\/strong>\uff1a1990 \u5e74\u4ee3\u306e\u6570\u5341\u4e07\u30da\u30fc\u30b8\u304b\u3089\u3001\u4eca\u65e5\u3067\u306f\u6570\u5341\u5104\u30da\u30fc\u30b8\u3078\u3068\u30af\u30ed\u30fc\u30eb\u898f\u6a21\u304c\u62e1\u5927\u3002\u9ad8\u5ea6\u306a\u30a4\u30f3\u30d5\u30e9\u30b9\u30c8\u30e9\u30af\u30c1\u30e3\u304c\u5fc5\u8981\u306b\u306a\u3063\u305f\u3002<\/li>\n<li><strong>\u7d71\u5408\u57fa\u76e4<\/strong>\uff1aGoogle \u306f\u3001\u8907\u6570\u306e\u30d7\u30ed\u30c0\u30af\u30c8\u30d5\u30a7\u30c3\u30c1\u30e3\u30fc\u3092 1 \u3064\u306e\u30af\u30ed\u30fc\u30e9\u30fc\u30d7\u30e9\u30c3\u30c8\u30d5\u30a9\u30fc\u30e0\u306b\u7d71\u5408\u3057\u3001robots.txt\u306e\u9075\u5b88\u3068\u793c\u5100\u6b63\u3057\u3055\u306e\u30eb\u30fc\u30eb\u3092\u7d71\u4e00\u7684\u306b\u65bd\u884c\u3057\u3066\u3044\u308b\u3002<\/li>\n<li><strong>\u5e2f\u57df\u5e45\u6700\u9069\u5316<\/strong>\uff1a\u6570\u30d0\u30a4\u30c8\u306e\u524a\u6e1b\u3067\u3082\u30b0\u30ed\u30fc\u30d0\u30eb\u898f\u6a21\u3067\u306f\u5927\u304d\u306a\u52b9\u679c\u3001\u305f\u3060\u3057\u30a4\u30f3\u30c7\u30c3\u30af\u30b9\u51e6\u7406\u304c\u6700\u5927\u30b3\u30b9\u30c8\u3002<\/li>\n<li><strong>\u30c7\u30e5\u30a2\u30eb\u53d6\u5f97\u30e2\u30fc\u30c9<\/strong>\uff1a\u30e6\u30fc\u30b6\u30fc\u8d77\u56e0\u306e\u30d5\u30a7\u30c3\u30c1\u306f\u901f\u5ea6\u3092\u512a\u5148\u3057\u3001robots.txt \u3092\u7121\u8996\u3059\u308b\u3053\u3068\u304c\u3042\u308b\u3002\u4e00\u65b9\u3001\u81ea\u52d5\u5316\u3055\u308c\u305f\u30af\u30ed\u30fc\u30e9\u30fc\u306f URL \u3092\u30ad\u30e5\u30fc\u306b\u5165\u308c\u3001\u30b5\u30a4\u30c8\u306e\u5236\u9650\u3092\u53b3\u5bc6\u306b\u9075\u5b88\u3059\u308b\u3002<\/li>\n<li><strong>\u4eca\u5f8c\u306e\u8ab2\u984c<\/strong>\uff1a\u3088\u308a\u591a\u304f\u306e\u30af\u30ed\u30fc\u30e9\u30fc\u3084AI\u30a8\u30fc\u30b8\u30a7\u30f3\u30c8\u306e\u767b\u5834\u306f\u7d99\u7d9a\u7684\u306a\u8ca0\u8377\u3092\u610f\u5473\u3057\u3001\u5171\u6709\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u3068\u30d7\u30ed\u30c8\u30b3\u30eb\u306e\u52b9\u7387\u6027\u304c\u4e0d\u53ef\u6b20\u3002<\/li>\n<\/ul>\n<p class=\"p60-t\">\u6700\u8fd1\u306f\u3001AI Overview\/AI Mode \u95a2\u9023\u306e\u8a18\u4e8b\u304c\u7d9a\u3044\u3066\u3044\u3066\u904e\u98df\u6c17\u5473\u3067\u3057\u305f\u3002<br \/>\u305f\u307e\u306b\u306f\u3001\u3053\u3046\u3044\u3063\u305f\u539f\u70b9\u306b\u5e30\u308b\u30c8\u30d4\u30c3\u30af\u3082\u7bb8\u4f11\u3081\u306b\u826f\u3044\u306e\u3067\u306f\u306a\u3044\u3067\u3057\u3087\u3046\u304b\u3002<\/p>\n<p>\u30dd\u30c3\u30c9\u30ad\u30e3\u30b9\u30c8\u5168\u4f53\u306f\u3053\u3061\u3089\u304b\u3089\u8074\u3051\u307e\u3059\u3002<\/p>\n<p><iframe loading=\"lazy\" width=\"560\" height=\"315\" src=\"https:\/\/www.youtube.com\/embed\/iGguggoNZ1E?si=yY_4k1h7Dw1dBq5u\" title=\"YouTube video player\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Google \u691c\u7d22\u30ea\u30ec\u30fc\u30b7\u30e7\u30f3\u30ba\u30c1\u30fc\u30e0\u306b\u3088\u308b\u30dd\u30c3\u30c9\u30ad\u30e3\u30b9\u30c8\u30b7\u30ea\u30fc\u30ba\u306e\u300eSearch Off the Record\u300f\u306e\u30a8\u30d4\u30bd\u30fc\u30c9 92 \u3067\u306f\u3001\u30a6\u30a7\u30d6\u30af\u30ed\u30fc\u30e9\u30fc\u306e\u6b74\u53f2\u3001\u8a2d\u8a08\u3001\u305d\u3057\u3066\u672a\u6765\u304c\u30c8\u30d4\u30c3\u30af\u306b\u306a\u3063\u305f\u3002<\/p>\n","protected":false},"author":1,"featured_media":15119,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[80,81],"class_list":["post-15118","post","type-post","status-publish","format-standard","has-post-thumbnail","category-google-seo","tag-crawler","tag-googlebot"],"_links":{"self":[{"href":"https:\/\/www.suzukikenichi.com\/blog\/wp-json\/wp\/v2\/posts\/15118","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.suzukikenichi.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.suzukikenichi.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.suzukikenichi.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.suzukikenichi.com\/blog\/wp-json\/wp\/v2\/comments?post=15118"}],"version-history":[{"count":0,"href":"https:\/\/www.suzukikenichi.com\/blog\/wp-json\/wp\/v2\/posts\/15118\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.suzukikenichi.com\/blog\/wp-json\/wp\/v2\/media\/15119"}],"wp:attachment":[{"href":"https:\/\/www.suzukikenichi.com\/blog\/wp-json\/wp\/v2\/media?parent=15118"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.suzukikenichi.com\/blog\/wp-json\/wp\/v2\/categories?post=15118"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.suzukikenichi.com\/blog\/wp-json\/wp\/v2\/tags?post=15118"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}