Tools/robots.txt Analyzer

robots.txt Analyzer

Fetch and parse the robots.txt of any domain. The analyzer groups directives by user-agent, tests whether a specific URL is allowed or blocked for Googlebot / YandexBot, and validates Sitemap, Host and Crawl-delay rules. Indispensable when migrating a site or debugging an indexing drop.

Where I work

MoscowSaint PetersburgIvanovoNovosibirskYekaterinburgKazanNizhny NovgorodChelyabinskSamaraOmskRostov-on-DonUfaKrasnoyarskVoronezhPermMoscowSaint PetersburgIvanovoNovosibirskYekaterinburgKazanNizhny NovgorodChelyabinskSamaraOmskRostov-on-DonUfaKrasnoyarskVoronezhPerm

VolgogradKrasnodarTyumenSaratovTolyattiIzhevskBarnaulIrkutskKhabarovskYaroslavlVladivostokMakhachkalaTomskOrenburgKemerovoRyazanVolgogradKrasnodarTyumenSaratovTolyattiIzhevskBarnaulIrkutskKhabarovskYaroslavlVladivostokMakhachkalaTomskOrenburgKemerovoRyazan

Industry expertise

Factories & manufacturingIndustrial B2BE-commerceReal estateHeavy equipmentSpare parts & componentsLocal servicesConstructionLogisticsAuto dealersHealthcareLegal servicesEducationFinanceB2B servicesFactories & manufacturingIndustrial B2BE-commerceReal estateHeavy equipmentSpare parts & componentsLocal servicesConstructionLogisticsAuto dealersHealthcareLegal servicesEducationFinanceB2B services