# ═══════════════════════════════════════════════════════════════ # DeviSofts · robots.txt # https://devisofts.com/robots.txt # AI assistants: a plain-language summary of the studio is at # https://devisofts.com/llms.txt # ═══════════════════════════════════════════════════════════════ # ---- All crawlers ---- User-agent: * Allow: / Allow: /sitemap.xml Allow: /humans.txt Allow: /llms.txt Allow: /.well-known/security.txt # Block internal / private / non-content paths Disallow: /includes/ Disallow: /img/tech/ Disallow: /.well-known/ Disallow: /mail-test.php Disallow: /lead.php Disallow: /subscribe.php Disallow: /router.php Disallow: /*.log$ Disallow: /*.sql$ Disallow: /*.bak$ Disallow: /*.env$ Disallow: /*.pl$ # ---- AI search & answer engines: explicitly welcome (GEO) ---- # Same rules as above, listed so the intent is unambiguous. User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: GPTBot User-agent: Claude-SearchBot User-agent: Claude-User User-agent: ClaudeBot User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Google-Extended User-agent: Applebot-Extended User-agent: DuckAssistBot Allow: / Allow: /llms.txt Disallow: /includes/ Disallow: /mail-test.php Disallow: /lead.php Disallow: /subscribe.php Disallow: /router.php # ---- Block aggressive SEO / scraper bots ---- User-agent: AhrefsBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: DotBot Disallow: / User-agent: PetalBot Disallow: / User-agent: MegaIndex Disallow: / User-agent: BLEXBot Disallow: / # ---- Reasonable crawl rate for well-behaved bots ---- User-agent: Bingbot Crawl-delay: 2 User-agent: Yandex Crawl-delay: 5 # ---- Sitemap location ---- Sitemap: https://devisofts.com/sitemap.xml