TildeClub
/
site
mirror of https://github.com/tildeclub/site.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
site/robots.txt

92 lines
1.5 KiB
Plaintext
Raw Permalink Normal View History

Add robots.txt This commit adds robots.txt entries to address web crawlers' accesses to user xwindows' web space. This patch is provided to Tilde.club under the terms of GNU General Public License, any version published by the Free Software Foundation.
2023-10-19 12:56:25 +00:00
User-Agent: MojeekBot
Allow: /~xwindows/
User-Agent: Qwantify
Allow: /~xwindows/
User-Agent: Wibybot
Allow: /~xwindows/
User-Agent: search.marginalia.nu
Allow: /~xwindows/
User-Agent: SearchMySiteBot
Allow: /~xwindows/
User-Agent: Duckduckbot
Allow: /~xwindows/
User-Agent: ia_archiver
Allow: /~xwindows/
User-Agent: Googlebot
Allow: /~xwindows/$
Allow: /~xwindows/index.html$
Disallow: /~xwindows/
User-Agent: bingbot
Allow: /~xwindows/$
Allow: /~xwindows/index.html$
Disallow: /~xwindows/
User-Agent: YandexBot
Allow: /~xwindows/$
Allow: /~xwindows/index.html$
Disallow: /~xwindows/
User-Agent: YandexFavicons
Disallow: /~xwindows/
User-Agent: MegaIndex.ru
Disallow: /~xwindows/
User-Agent: Amazonbot
Disallow: /~xwindows/
User-Agent: Linespider
Disallow: /~xwindows/
User-Agent: Bytespider
Disallow: /~xwindows/
User-Agent: CCBot
Disallow: /~xwindows/
User-Agent: Neevabot
Disallow: /~xwindows/
User-Agent: PetalBot
Disallow: /~xwindows/
User-Agent: SemrushBot
Disallow: /~xwindows/
User-Agent: AhrefsBot
Disallow: /~xwindows/
User-Agent: DataForSeoBot
Disallow: /~xwindows/
User-Agent: dotbot
Disallow: /~xwindows/
User-Agent: Barkrowler
Disallow: /~xwindows/
User-Agent: MJ12bot
Disallow: /~xwindows/
User-Agent: BuiltWith
Disallow: /~xwindows/
User-Agent: webprosbot
Disallow: /~xwindows/
User-Agent: Dataprovider
Disallow: /~xwindows/
User-Agent: *
Allow: /~xwindows/$
Allow: /~xwindows/index.html$
Disallow: /~xwindows/