# Public pages and their assets are available to search and AI search crawlers, # including Googlebot, Bingbot and OAI-SearchBot through the wildcard group. User-agent: * Allow: / Disallow: /api/ Disallow: /.netlify/ Disallow: /admin/ Disallow: /404.html Disallow: /_headers Sitemap: https://rayanfm.net/sitemap.xml User-agent: AhrefsBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: DotBot Disallow: / User-agent: MJ12bot Disallow: /