What is Applebot? How to allow or block Apple's crawler

Applebot is Apple's agent for AI search and live answers. Apple's crawler powering Siri and Spotlight suggestions.

Apple documents Applebot thoroughly and honors robots.txt directives.

Add "User-agent: Applebot" followed by "Disallow: /" to your robots.txt file. The change takes effect the next time the bot fetches your robots.txt.

No. Applebot is separate from Googlebot, which handles Google Search indexing. Blocking Applebot has no effect on your traditional search rankings, but it does remove your pages from the AI answers Apple's assistant serves to its users.

Search your server access logs for the string "Applebot" — for example: grep -i "Applebot" /var/log/nginx/access.log | wc -l. Our free AI Bot Log Analyzer does this in your browser: paste a log file and it counts hits per AI crawler, including Applebot, with per-path breakdowns.

You don't — Applebot isn't a tool you run. It's Apple's own crawler, operated by Apple, that visits your site from their infrastructure. The only control you have over it is whether you allow or block it, via robots.txt or a server rule. If you're looking to crawl other sites yourself, you'd write your own crawler or use a crawling library; sending "Applebot" as your user agent would be impersonating Apple.

No, and Apple says so directly. Apple's documentation states that Applebot does not follow crawl-delay, and that it adjusts its own rate instead to limit the load it puts on a site. Crawl-delay was never part of the original robots.txt specification, and Google's decision to ignore it set the pattern most crawlers follow. To actually slow Applebot down, rate-limit it at the server or CDN — a Cloudflare rate-limiting rule or an nginx limit_req zone matched on the user agent is enforced, where a crawl-delay line is not.

Apple states Applebot honors robots.txt, so if you are still seeing hits the cause is usually one of four things rather than the bot misbehaving. First, robots.txt is cached — Apple may be working from a copy fetched up to 24 hours before your change. Second, the rule may not match: robots.txt user-agent matching is on a prefix of the token, and a typo or a trailing character breaks it silently. Third, the block may sit under a different user-agent group than the one this bot reads, since a bot obeys only the most specific group that matches it, not the wildcard group as well. Fourth, the traffic may be something else sending Applebot as its user agent, which anything can do.

Treat it as no unless Apple says otherwise. Rendering JavaScript costs an order of magnitude more than fetching HTML, and most AI crawlers — unlike Googlebot, which runs a full headless Chrome — read the raw HTML response and stop there. The practical consequence: anything your page loads client-side after the initial response is likely invisible to Applebot. If your main content is client-rendered, server-render it or pre-render it so the text exists in the first response.

Apple publishes them as JSON at https://search.developer.apple.com/applebot.json, updated as their infrastructure changes. Fetch that file rather than hard-coding the prefixes, and match the request IP against it to confirm a visit is genuinely Applebot. This is the only reliable check — the user-agent header is self-declared, so a scraper can send "Applebot" while coming from anywhere.

There is no fixed schedule. Applebot serves live answers, so its crawl rate tracks demand — pages that come up in Apple conversations get fetched more often, and a site nobody asks about may see it rarely or never. Frequency also rises with how often your pages change and how many links point at them. The only way to know for your own site is to measure it: grep your access log for the user agent, or paste the log into our AI Bot Log Analyzer, which breaks hits down by date and path in your browser.

Apple publishes it at https://support.apple.com/en-us/119829. That page is the authoritative source for the user-agent string and Apple's stated crawling policy, and Apple also publishes Applebot's IP ranges as JSON at https://search.developer.apple.com/applebot.json so you can verify requests rather than trusting the header.