# miniscaping.com robots policy (2026-11-04; updated 2026-11-07, 2026-11-11) # Search engines: welcome. AI crawlers (well-behaved ones): welcome. # Bulk scrapers: blocked at the edge (UA list + fail2ban rate bans). # # 2026-11-07: combinatorial filter permutations + /pixel.gif disallowed for # EVERYONE. ClaudeBot ramped to ~900k req/day after the 2026-11-04 unblock by # recursively crawling filter permutations; filter URLs have no SEO value. # 2026-11-11: rule generalized to /*/filter — GPTBot ramped to ~170k req/day # walking sibling patterns (/explore/*/miniscapes/filter, /explore/*/filter, # /experience/.../filter) not covered by the original /explore/*/inspirations/filter. # NOTE: for a named agent below, only its own group applies (most specific # group wins, RFC 9309) — so the Disallows are repeated in each group. User-agent: * Disallow: /api/ Disallow: /dashboard Disallow: /auth/ Disallow: /assets/uploads/original/ Disallow: /assets/cache/ Disallow: /pixel.gif Disallow: /*/filter # Search engines (explicitly welcome) User-agent: Googlebot Disallow: /pixel.gif Disallow: /*/filter Allow: / User-agent: Bingbot Disallow: /pixel.gif Disallow: /*/filter Allow: / User-agent: DuckDuckBot Disallow: /pixel.gif Disallow: /*/filter Allow: / # AI crawlers (well-behaved: honor robots.txt) User-agent: GPTBot Disallow: /pixel.gif Disallow: /*/filter Allow: / User-agent: ClaudeBot Disallow: /pixel.gif Disallow: /*/filter Allow: / User-agent: Amazonbot Disallow: /pixel.gif Disallow: /*/filter Allow: / User-agent: Applebot Disallow: /pixel.gif Disallow: /*/filter Allow: / Sitemap: https://miniscaping.com/sitemap.xml