{"id":14746,"date":"2025-01-30T06:55:16","date_gmt":"2025-01-29T21:55:16","guid":{"rendered":"https:\/\/www.suzukikenichi.com\/blog\/?p=14746"},"modified":"2025-01-30T06:51:29","modified_gmt":"2025-01-29T21:51:29","slug":"why-does-search-console-report-the-date-when-googlebot-crawled-blocked-pages","status":"publish","type":"post","link":"https:\/\/www.suzukikenichi.com\/blog\/why-does-search-console-report-the-date-when-googlebot-crawled-blocked-pages\/","title":{"rendered":"robots.txt\u3067\u30d6\u30ed\u30c3\u30af\u3057\u305f\u30da\u30fc\u30b8\u306e\u30af\u30ed\u30fc\u30eb\u65e5\u304cSearch Console\u306b\u30ec\u30dd\u30fc\u30c8\u3055\u308c\u308b\u306e\u306f\u306a\u305c\uff1f"},"content":{"rendered":"<p class=\"p20-b al-r small\">[\u30ec\u30d9\u30eb: \u4e2d\u7d1a]<\/p>\n<p class=\"p60-b\">robots.txt \u3067\u30af\u30ed\u30fc\u30eb\u3092\u30d6\u30ed\u30c3\u30af\u3057\u305f URL \u306b\u3082\u524d\u56de\u306e\u30af\u30ed\u30fc\u30eb\u65e5\u304c Search Console \u306b\u8a18\u9332\u3055\u308c\u307e\u3059\u3002<br \/>\u3057\u304b\u3057\u3001Googlebot \u306f\u305d\u306e URL \u3092\u5b9f\u969b\u306b\u306f\u30af\u30ed\u30fc\u30eb\u3057\u3066\u306f\u3044\u307e\u305b\u3093\u3002<\/p>\n<h2>\u30af\u30ed\u30fc\u30eb\u8a66\u884c\u5931\u6557\u306e\u65e5\u4ed8<\/h2>\n<blockquote><p>\u9577\u3044\u671f\u9593\u306b\u308f\u305f\u3063\u3066 robots.txt \u3067\u30af\u30ed\u30fc\u30eb\u3092\u7981\u6b62\u3057\u3066\u3044\u305f URL \u304c\u3042\u308b\u304c\u3001Search Console \u306e\u30da\u30fc\u30b8\u30a4\u30f3\u30c7\u30c3\u30af\u30b9\u30ec\u30dd\u30fc\u30c8\u306e\u300crobots.txt \u306b\u3088\u308a\u30d6\u30ed\u30c3\u30af\u3055\u308c\u307e\u3057\u305f\u300d\u3067\u306f\u300c\u524d\u56de\u306e\u30af\u30ed\u30fc\u30eb\u300d\u306e\u65e5\u4ed8\u304c\u5b9a\u671f\u7684\u306b\u66f4\u65b0\u3055\u308c\u3066\u3044\u308b\u3002\u3057\u304b\u3057\u3001\u30b5\u30fc\u30d0\u30fc\u3084 CDN \u306e\u30ed\u30b0\u306b\u306f Googlebot \u306e\u30a2\u30af\u30bb\u30b9\u306f\u8a18\u9332\u3055\u308c\u3066\u3044\u306a\u3044\u3002<\/p>\n<p>\u3053\u308c\u306f\u3069\u3046\u3044\u3046\u3053\u3068\u306a\u306e\u304b\uff1f<\/p><\/blockquote>\n<p><a href=\"https:\/\/bsky.app\/profile\/fearless-shultz.bsky.social\/post\/3lgsa6tne7s2n\">\u3053\u3046\u3057\u305f\u8da3\u65e8\u306e\u8cea\u554f<\/a>\u3092 Google \u306e John Mueller\uff08\u30b8\u30e7\u30f3\u30fb\u30df\u30e5\u30fc\u30e9\u30fc\uff09\u304c\u5c0b\u306d\u3089\u308c\u307e\u3057\u305f\u3002<\/p>\n<p>\u7aef\u7684\u306b\u72b6\u6cc1\u3092\u307e\u3068\u3081\u308b\u3068\u3053\u3046\u306a\u308a\u307e\u3059\u3002<\/p>\n<ul>\n<li>robots.txt \u3067\u30af\u30ed\u30fc\u30eb\u3092\u30d6\u30ed\u30c3\u30af\u3057\u305f URL \u304c Search Console \u3067\u306f\u30af\u30ed\u30fc\u30eb\u3055\u308c\u3066\u3044\u308b\u3088\u3046\u306b\u898b\u3048\u308b\uff08\u30af\u30ed\u30fc\u30eb\u65e5\u304c\u5b9a\u671f\u7684\u306b\u66f4\u65b0\u3055\u308c\u3066\u3044\u308b\u304b\u3089\uff09<\/li>\n<li>\u30b5\u30fc\u30d0\u30fc\u30ed\u30b0\u306b\u306f Googlebot \u304c\u30af\u30ed\u30fc\u30eb\u3057\u305f\u5f62\u8de1\u306f\u306a\u3044<\/li>\n<\/ul>\n<p class=\"p30-t\">robots.txt \u3067\u30d6\u30ed\u30c3\u30af\u3057\u305f\u30da\u30fc\u30b8\u3067\u3082 Search Console \u3067\u306f\u78ba\u304b\u306b\u30af\u30ed\u30fc\u30eb\u65e5\u304c\u8a18\u9332\u3055\u308c\u3066\u3044\u307e\u3059\u3002<\/p>\n<picture><source srcset=\"https:\/\/www.suzukikenichi.com\/images\/date-blocked-page-crawled.avif\" type=\"image\/avif\" \/><img decoding=\"async\" class=\"bdr\" itemprop=\"image\" loading=\"lazy\" src=\"https:\/\/www.suzukikenichi.com\/images\/date-blocked-page-crawled.webp\" alt=\"robots.txt \u306b\u3088\u308a\u30d6\u30ed\u30c3\u30af\u3055\u308c\u307e\u3057\u305f\u30da\u30fc\u30b8\u306e\u524d\u56de\u306e\u30af\u30ed\u30fc\u30eb\u65e5\" width=\"855\" height=\"859\" \/>\n<\/picture>\n<p class=\"p30-b\">\u7981\u6b62\u3055\u308c\u3066\u3044\u308b\u306e\u306b\u30af\u30ed\u30fc\u30eb\u3057\u305f\u3068\u3044\u3046\u3053\u3068\u3067\u3057\u3087\u3046\u304b\uff1f<\/p>\n<p>\u30df\u30e5\u30fc\u30e9\u30fc\u6c0f\u306f\u6b21\u306e\u3088\u3046\u306b<a href=\"https:\/\/bsky.app\/profile\/johnmu.com\/post\/3lgsdkoy37k2j\">\u56de\u7b54<\/a>\u3057\u307e\u3057\u305f\u3002<\/p>\n<blockquote><p>\u3082\u3057\u305d\u3046\u3057\u305f URL \u304c\u300c\u30d6\u30ed\u30c3\u30af\u3055\u308c\u307e\u3057\u305f\u300d\u3068\u8868\u793a\u3055\u308c\u3066\u3044\u308b\u5834\u5408\u3001\u305d\u308c\u306f\u57fa\u672c\u7684\u306b\u30af\u30ed\u30fc\u30eb\u8a66\u884c\u306e\u5931\u6557\u3092\u610f\u5473\u3059\u308b\u3002\u3064\u307e\u308a\u3001Google \u306e\u30b7\u30b9\u30c6\u30e0\u306f\u30af\u30ed\u30fc\u30eb\u3057\u3088\u3046\u3068\u3057\u305f\u3082\u306e\u306e\u3001robots.txt \u306b\u3088\u3063\u3066\u30d6\u30ed\u30c3\u30af\u3055\u308c\u305f\u305f\u3081\u3001\u5b9f\u969b\u306e\u30af\u30ed\u30fc\u30eb\u306f\u884c\u308f\u308c\u3066\u3044\u306a\u3044\u3002\u305d\u306e\u305f\u3081\u3001\u30b5\u30fc\u30d0\u30fc\u30ed\u30b0\u306b\u306f\u8a18\u9332\u3055\u308c\u306a\u3044\uff08\u307e\u305f\u306f\u8a18\u9332\u3055\u308c\u308b\u3079\u304d\u3067\u306f\u306a\u3044\uff09\u3068\u3044\u3046\u3053\u3068\u306b\u306a\u308b\u3002<\/p><\/blockquote>\n<p>\u300crobots.txt \u306b\u3088\u308a\u30d6\u30ed\u30c3\u30af\u3055\u308c\u307e\u3057\u305f\u300d\u3067\u8a18\u9332\u3055\u308c\u308b\u524d\u56de\u306e\u30af\u30ed\u30fc\u30eb\u65e5\u306f\u3001<mark>\u30af\u30ed\u30fc\u30eb<strong>\u3057\u305f<\/strong>\u65e5\u3067\u306f\u306a\u304f\u30af\u30ed\u30fc\u30eb<strong>\u3057\u3088\u3046\u3068\u3057\u305f\uff08\u3067\u3082\u3067\u304d\u306a\u304b\u3063\u305f\uff09<\/strong>\u65e5<\/mark>\u3092\u610f\u5473\u3059\u308b\u3068\u3044\u3046\u3053\u3068\u3067\u3059\u3002<\/p>\n<p>\u5b9f\u969b\u306b\u306f\u30af\u30ed\u30fc\u30eb\u3057\u3066\u3044\u306a\u3044\u306e\u3067\u3059\u304b\u3089\u3001\u30b5\u30fc\u30d0\u30fc\u30ed\u30b0\u306b\u306f\u5f53\u7136\u73fe\u308c\u307e\u305b\u3093\u3002<\/p>\n<p class=\"p40-b\">\u3053\u308c\u306f\u3001robots.txt \u3067\u30af\u30ed\u30fc\u30eb\u3092\u6b63\u5e38\u306b\u30d6\u30ed\u30c3\u30af\u3067\u304d\u3066\u3044\u308c\u3070\u3001\u305f\u3068\u3048 Search Console \u3067\u30af\u30ed\u30fc\u30eb\u65e5\u304c\u66f4\u65b0\u3055\u308c\u3066\u3044\u305f\u3068\u3057\u3066\u3082\u3001<a href=\"https:\/\/www.suzukikenichi.com\/blog\/what-is-really-crawl-budget\/\">\u30af\u30ed\u30fc\u30eb\u30d0\u30b8\u30a7\u30c3\u30c8<\/a>\u3092\u6d88\u8cbb\u3057\u306a\u3044\u3068\u3044\u3046\u3053\u3068\u306b\u3082\u7e4b\u304c\u308a\u307e\u3059\u3002<\/p>\n<blockquote class=\"bluesky-embed\" data-bluesky-uri=\"at:\/\/did:plc:lfukbggn4kz2dwhnr66lei7x\/app.bsky.feed.post\/3lgsa6tne7s2n\" data-bluesky-cid=\"bafyreifmrhjw4ksemwu3voaj3zvqkuguvzm5i5xc2q56t3buwqctf7gwei\">\n<p lang=\"en\">Hi @johnmu.com I\u2019ve a set of URLs that have been disallowed for years which are showing regular Last Crawled date updates in GSC Blocked by robots.txt report but there is no record in our Apache logs or Cloudflare of HTTP requests for these URLs or them blocked from being logged in some way<\/p>\n<p>&mdash; MikeO (<a href=\"https:\/\/bsky.app\/profile\/did:plc:lfukbggn4kz2dwhnr66lei7x?ref_src=embed\">@fearless-shultz.bsky.social<\/a>) <a href=\"https:\/\/bsky.app\/profile\/did:plc:lfukbggn4kz2dwhnr66lei7x\/post\/3lgsa6tne7s2n?ref_src=embed\">January 28, 2025 at 7:20 PM<\/a><\/p><\/blockquote>\n<p><script defer src=\"https:\/\/embed.bsky.app\/static\/embed.js\" charset=\"utf-8\"><\/script><\/p>\n<blockquote class=\"bluesky-embed\" data-bluesky-uri=\"at:\/\/did:plc:4cv34f5o756rx377sm3mhrhm\/app.bsky.feed.post\/3lgsdkoy37k2j\" data-bluesky-cid=\"bafyreidr2aadcbmjivjhaykeoxxwq3xqf7c5qvhuappmaivhltf37ortia\">\n<p lang=\"en\">If they&#x27;re disallowed, and showing as &quot;blocked by robots&quot;, then it&#x27;s basically a failed crawl attempt &#8211; our systems wanted to crawl, but couldn&#x27;t (because of robots.txt). You wouldn&#x27;t (shouldn&#x27;t) see it in your server logs.<\/p>\n<p>&mdash; John Mueller (<a href=\"https:\/\/bsky.app\/profile\/did:plc:4cv34f5o756rx377sm3mhrhm?ref_src=embed\">@johnmu.com<\/a>) <a href=\"https:\/\/bsky.app\/profile\/did:plc:4cv34f5o756rx377sm3mhrhm\/post\/3lgsdkoy37k2j?ref_src=embed\">January 28, 2025 at 8:20 PM<\/a><\/p><\/blockquote>\n","protected":false},"excerpt":{"rendered":"<p>robots.txt \u3067\u30af\u30ed\u30fc\u30eb\u3092\u30d6\u30ed\u30c3\u30af\u3057\u305f URL \u306b\u3082\u524d\u56de\u306e\u30af\u30ed\u30fc\u30eb\u65e5\u304c Search Console \u306b\u8a18\u9332\u3055\u308c\u308b\u3002\u3057\u304b\u3057\u3001Googlebot \u306f\u305d\u306e URL \u3092\u5b9f\u969b\u306b\u306f\u30af\u30ed\u30fc\u30eb\u3057\u3066\u306f\u3044\u306a\u3044\u3002<\/p>\n","protected":false},"author":1,"featured_media":14747,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[29],"tags":[62,1301],"class_list":["post-14746","post","type-post","status-publish","format-standard","has-post-thumbnail","category-google-webmaster-tools","tag-robotstxt","tag-crawl-budget"],"_links":{"self":[{"href":"https:\/\/www.suzukikenichi.com\/blog\/wp-json\/wp\/v2\/posts\/14746","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.suzukikenichi.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.suzukikenichi.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.suzukikenichi.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.suzukikenichi.com\/blog\/wp-json\/wp\/v2\/comments?post=14746"}],"version-history":[{"count":0,"href":"https:\/\/www.suzukikenichi.com\/blog\/wp-json\/wp\/v2\/posts\/14746\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.suzukikenichi.com\/blog\/wp-json\/wp\/v2\/media\/14747"}],"wp:attachment":[{"href":"https:\/\/www.suzukikenichi.com\/blog\/wp-json\/wp\/v2\/media?parent=14746"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.suzukikenichi.com\/blog\/wp-json\/wp\/v2\/categories?post=14746"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.suzukikenichi.com\/blog\/wp-json\/wp\/v2\/tags?post=14746"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}