屏蔽AhrefsBot蜘蛛的方法

前言

最近，查看网站访问日志，发现大量来自 AhrefsBot 的爬取记录：

51.222.253.8 - - [27/Jan/2022:05:35:04 +0000] "GET / HTTP/2.0" 200 5862 "-" "Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)"
51.222.253.19 - - [27/Jan/2022:05:57:51 +0000] "GET / HTTP/2.0" 200 5762 "-" "Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)"
51.222.253.7 - - [27/Jan/2022:06:10:48 +0000] "GET /c_docker HTTP/2.0" 200 2029 "-" "Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)"
51.222.253.3 - - [27/Jan/2022:06:25:55 +0000] "GET /p_git-ignore-not-working HTTP/2.0" 200 1951 "-" "Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)"
51.222.253.13 - - [27/Jan/2022:06:40:11 +0000] "GET /p_chrome-plugin-develop HTTP/2.0" 200 2520 "-" "Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)"
51.222.253.18 - - [27/Jan/2022:06:55:10 +0000] "GET /tags/eggjs HTTP/2.0" 200 1152 "-" "Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)"
51.222.253.4 - - [27/Jan/2022:07:10:34 +0000] "GET /c_nunjucks HTTP/2.0" 200 1122 "-" "Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)"
51.222.253.4 - - [27/Jan/2022:07:18:34 +0000] "GET /p_node-puppeteer-screenshot HTTP/2.0" 200 2500 "-" "Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)"
51.222.253.4 - - [27/Jan/2022:07:26:27 +0000] "GET /p_nunjucks-guide HTTP/2.0" 200 10944 "-" "Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)"
51.222.253.14 - - [27/Jan/2022:07:35:09 +0000] "GET /tags/%E6%97%A0%E5%A4%B4%E6%B5%8F%E8%A7%88%E5%99%A8 HTTP/2.0" 200 961 "-" "Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)"
51.222.253.7 - - [27/Jan/2022:07:44:21 +0000] "GET /tags/puppeteer HTTP/2.0" 200 947 "-" "Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)"
51.222.253.5 - - [27/Jan/2022:07:54:02 +0000] "GET /c_rich-text-editor HTTP/2.0" 200 1344 "-" "Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)"
51.222.253.3 - - [27/Jan/2022:08:03:55 +0000] "GET /tags/%E6%96%87%E4%BB%B6%E6%9F%A5%E6%89%BE HTTP/2.0" 200 1010 "-" "Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)"
51.222.253.5 - - [27/Jan/2022:08:13:56 +0000] "GET /tags/%E7%BC%93%E5%AD%98 HTTP/2.0" 200 930 "-" "Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)"

AhrefsBot 是什么？

AhrefsBot 是一个 Web 爬虫程序，为 Ahrefs 在线营销工具集提供据。对我们来说没有任何意义，屏蔽它！

如何禁止 AhrefsBot 爬取网页？

方法很简单，要禁止某个搜索爬虫，在 robots.txt 文件中设置即可：

robots.txt

User-agent: AhrefsBot
Disallow: /

以上就是屏蔽 AhrefsBot 的方法。

本文地址：https://www.tides.cn/p_seo-disallow-robot-ahrefsbot