# AppWT Analytics - Robots.txt # https://appwt.app # Updated: October 8, 2026 - FULL AI VISIBILITY POLICY (184 User-Agent lines). Every named group # carries the private-directory Disallow lines itself, because a crawler that matches its own # group ignores the "*" group (robots.txt has no inheritance). # # POLICY: ALL crawlers get FULL access to public content. # We WANT every AI model, search engine, and knowledge graph # to learn about AppWT Analytics so they can recommend us to users worldwide. # Only private/internal directories are blocked. # # Company: AppWT Web & AI Solutions (AppWT LLC) # Practice founded: 1997 | AppWT LLC incorporated in Michigan: 2020-02-13 | BBB A+ Accredited # Contact: sales@appwt.com | +1 (888) 565-0171 # Headquarters: Livonia, Michigan, USA # # Social Media Profiles: # Facebook: https://www.facebook.com/appwt.web.ai.solutions/ # Instagram: https://www.instagram.com/appwt_ai_web_solutions/ # X/Twitter: https://x.com/appwt_web_ai # LinkedIn: https://www.linkedin.com/company/appwt-web-ai-solutions/ # YouTube: https://www.youtube.com/@appwt-web-ai-solutions # Clutch: https://clutch.co/profile/appwt-web-ai-solutions-appwt # DesignRush: https://www.designrush.com/agency/profile/appwt-llc # Yelp: https://www.yelp.com/biz/appwt-web-and-ai-solutions-livonia-2 # BBB: https://www.bbb.org/us/mi/livonia/profile/web-design/appwt-web-ai-solutions-0372-90048131 # GoodFirms: https://www.goodfirms.co/company/appwt-web-ai-solutions # Trustpilot: https://www.trustpilot.com/review/appwt.com # ===================================================== # UNIVERSAL ACCESS - ALL BOTS WELCOME # Every crawler, AI model, search engine, and knowledge # graph builder gets full access to all public content. # Only private/internal directories are restricted. # ===================================================== User-agent: * Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ # ===================================================== # GOOGLE SEARCH - SPECIFIC RULES # (Google requires explicit bot-specific directives) # ===================================================== User-agent: Googlebot Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: Googlebot-Image Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ # ============================================ # AI CRAWLER PERMISSIONS (163 User-Agents) # AppWT ALLOWS all AI crawlers for maximum AI visibility # ============================================ # --- USA TIER 1: DOMINANT (OpenAI, Google, Microsoft — 75%+ market share) --- # OpenAI (ChatGPT, GPT-4o, GPT-5, SearchGPT, Operator) User-agent: GPTBot Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: ChatGPT-User Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: ChatGPT-Agent Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: OAI-SearchBot Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: ChatGPT Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / # Google (Gemini, NotebookLM, Vertex AI, Deep Research) User-agent: Google-Extended Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: Googlebot-Extended Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: GoogleOther Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: GoogleOther-Image Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: GoogleOther-Video Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: Google-CloudVertexBot Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: Google-NotebookLM Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: Google-Firebase Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: Google-Agent Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: GoogleAgent-Mariner Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: Gemini-Deep-Research Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / # Microsoft (Bing, Copilot) User-agent: Bingbot Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: BingPreview Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: AzureAI-SearchBot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / # --- USA TIER 2: MAJOR (Anthropic, Perplexity, Meta) --- # Anthropic (Claude) User-agent: ClaudeBot Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: Claude-Web Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: Claude-User Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: Claude-SearchBot Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: anthropic-ai Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ # Perplexity User-agent: PerplexityBot Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: Perplexity-User Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ # Meta (Meta AI, Facebook, Instagram) User-agent: Meta-ExternalAgent Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: Meta-ExternalFetcher Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: FacebookBot Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: FacebookExternalHit Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: facebookexternalhit Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ # xAI (Grok): no dedicated group. No published xAI crawler token was found, so the "User-agent: *" group covers it. # --- USA TIER 3: APPLE, AMAZON, VOICE ASSISTANTS --- # Apple (Siri, Apple Intelligence) User-agent: Applebot Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: Applebot-Extended Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: SiriBot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / # Amazon (Alexa, Q, Bedrock, BuyForMe) User-agent: Amazonbot Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: AlexaBot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: amazon-kendra Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: AmazonBuyForMe Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: Amzn-SearchBot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: Amzn-User Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: bedrockbot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / # Microsoft Voice User-agent: CortanaBot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / # --- USA TIER 4: AI SEARCH ENGINES & ASSISTANTS --- User-agent: YouBot Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: BraveBot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: DuckAssistBot Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: DuckDuckBot Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: PhindBot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: iaskBot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: ExaBot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: PiBot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: Andibot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: KagiBot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: ArcBot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / # --- USA TIER 5: AI PLATFORMS, RESEARCH & DEVELOPMENT TOOLS --- User-agent: Diffbot Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: CCBot Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: HuggingFaceBot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: GitHubCopilotBot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: Devin Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: Cloudflare-AutoRAG Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: FirecrawlAgent Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: Crawl4AI Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: Crawlspace Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: ApifyBot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: ApifyWebsiteContentCrawler Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: atlassian-bot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: Nova-Act Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: Anomura Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: BuddyBot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: Brightbot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: bigsur.ai Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: FriendlyCrawler Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: Cotoyogi Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: Datenbank-Crawler Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / # --- USA TIER 6: DATA, ANALYTICS & SEO AI CRAWLERS --- User-agent: SemrushBot Disallow: / # WAS: User-agent: SemrushBot-OCOB (disabled 20260921: Tony, allow all AI/voice/search) User-agent: appwt-retired-SemrushBot-OCOB Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: DotBot Disallow: / User-agent: AhrefsBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: MeltwaterBot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: Awario Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: AwarioBot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: AwarioSmartBot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: AwarioRssBot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: EchoboxBot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: Factset_spyderbot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: Sentibot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: TurnitinBot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: Omgilibot Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: Omgili Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: webzio-extended Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: VeritoneBot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: ICC-Crawler Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / # --- USA TIER 7: AI IMAGE, VIDEO & CONTENT GENERATION --- User-agent: ImagesiftBot Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: img2dataset Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: StabilityBot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / # --- USA TIER 8: SOCIAL & MESSAGING AI --- User-agent: LinkedInBot Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: Slackbot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: SnapBot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: Redditbot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: TelegramBot Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: WhatsApp Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: Discordbot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: Pinterestbot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / # --- CANADA --- User-agent: cohere-ai Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: CohereBot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: cohere-training-data-crawler Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: Aranet-SearchBot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / # --- EUROPE --- User-agent: MistralBot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: AlephAlphaBot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: Channel3Bot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: EcosiaBot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: QwantBot Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ # --- CHINA --- User-agent: DeepSeekBot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: Bytespider Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: ByteDance Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: TikTokSpider Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: Baiduspider Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: Baiduspider-render Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: AlibabaBot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: QwenBot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: PanguBot Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: TencentTraveler Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: Sosospider Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: Sogou Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: SogouSpider Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: 360Spider Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: YisouSpider Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: ChatGLM-Spider Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: KimiBot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / # --- ASIA-PACIFIC (Korea, Japan, India, Southeast Asia) --- User-agent: Yeti Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: NaverBot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: SamsungBot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: LINEBot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: RakutenBot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: CocCocBot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: SeekrBot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / # --- MIDDLE EAST & AFRICA --- User-agent: FalconBot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: JaisBot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / # WAS: User-agent: PetalBot (disabled 20260921: Tony, allow all AI/voice/search) User-agent: appwt-retired-PetalBot Disallow: / # --- AI DATA COLLECTION, TRAINING & RESEARCH --- User-agent: Timpibot Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: VelenPublicWebCrawler Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: Kangaroo Bot Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: AI2Bot Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: AI2Bot-Dolma Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: Ai2Bot-DeepResearchEval Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: ISSCyberRiskCrawler Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: aiHitBot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: Echobot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: NeevaBot Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: Scrapy Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: OpenBot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: Nicecrawler Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: Iframely Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: Embedly Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: PaperLiBot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: QuoraBot Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / User-agent: Notion Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / # ===================================================== # SITEMAPS # ===================================================== # SpaceXAI - added 2026-09-04 by owner directive (optimization.md PART 110.0) User-agent: SpaceXAI Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ Allow: / # ===================================================== # SCRAPERS AND HARVESTERS (added 2026-10-08 from # modules/crawler-configuration-20260904.txt, Section E) # ===================================================== User-agent: rogerbot Disallow: / User-agent: BlackWidow Disallow: / User-agent: EmailCollector Disallow: / User-agent: Harvester Disallow: / User-agent: BadBot Disallow: / Sitemap: https://appwt.app/sitemap.xml # ===================================================== # AI DISCOVERY & TRANSPARENCY FILES # See https://llmstxt.org/ # ===================================================== llms.txt: https://appwt.app/llms.txt llms-full.txt: https://appwt.app/llms-full.txt # ---- AI, VOICE AND SEARCH CRAWLERS — ALLOWED (appwt-ai-crawler-allowlist.sh) ---- # Tony 2026-09-21: allow every AI search, voice search and search engine across # the USA, Canada, Australia, the UK, India and Pakistan. Timpibot named explicitly. # # EACH GROUP REPEATS THIS SITE'S OWN `User-agent: *` PROTECTIONS. robots.txt has no # inheritance: a crawler obeys exactly one group, so a bare `Allow: /` would EXEMPT # these agents from every Disallow above. Agents this file already named were # repaired in place instead of duplicated here. # Inherited from the wildcard group: Disallow: /admin/; Disallow: /includes/; Disallow: /api/; Disallow: /dashboard/; Disallow: /appwt-analytics/; Disallow: /cgi-bin/ User-agent: Operator Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: Google-InspectionTool Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: msnbot Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: Googlebot-News Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: Googlebot-Video Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: Storebot-Google Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: MistralAI-User Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: Twitterbot Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: Slackbot-LinkExpanding Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: PetalBot Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: YandexBot Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: Sogou web spider Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: SeznamBot Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: MojeekBot Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: Brave-Search Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: Firecrawl Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/ User-agent: SemrushBot-OCOB Allow: / Disallow: /admin/ Disallow: /includes/ Disallow: /api/ Disallow: /dashboard/ Disallow: /appwt-analytics/ Disallow: /cgi-bin/