QuikSEOBot Crawler User Agent

How QuikSEOBot Identifies Itself

Every request of a crawl carries the QuikSEOBot user agent below and comes from the QuikSEO server, never from your visitors' browsers.

QuikSEOBot/0.1 (+https://quikseo.com/docs/crawler)

• It sends no cookies, no login data and no authorization headers.
• It requests only http and https addresses on ports 80 and 443, and never private or internal addresses.
• It reads at most 2 MB of HTML per page and reports a larger page as too large instead of guessing.

Control QuikSEOBot in robots.txt

QuikSEOBot reads your robots.txt once at the start of each crawl and obeys the group for QuikSEOBot, or the * group when there is none, following RFC 9309.

• A robots.txt that answers 404 or another 4xx status allows every page.
• A robots.txt that answers 5xx, times out or cannot be reached stops the crawl before any page is requested.
• A blocked URL is never requested; the report lists it with the robots.txt line that blocked it.

Keep QuikSEOBot Out of a Section

A Disallow line in a QuikSEOBot group keeps the crawler out of that path while other crawlers keep their own rules.

User-agent: QuikSEOBot
Disallow: /checkout/
Disallow: /account/

Keep QuikSEOBot Out of the Whole Site

A single Disallow of the root stops every crawl of the site; audits of single pages are not crawls and are not affected.

User-agent: QuikSEOBot
Disallow: /

What a Crawl Visits

A crawl visits the pages of one website in order of click depth from the start page, so the pages closest to the start page are audited first.

1. robots.txt of the website.
2. The sitemaps that robots.txt or the website settings declare, or /sitemap.xml when none is declared, up to 20 files.
3. The start page, then the pages it links to, then the pages those link to, up to 10 clicks deep.
4. Pages found only in the sitemap, after the linked pages.

Some URLs are recorded but not requested:

• links marked rel="nofollow";
• links to files such as PDF, images, archives, stylesheets and scripts;
• links to other websites, including other subdomains;
• URLs beyond the page limit of the crawl, listed as not reached.

A redirect is not followed on the spot: its target joins the queue like any other URL, so no page is requested twice.

For the same reason the Redirect Chain rule cannot see a chain inside a crawl and reports every crawled page as passing it; audit a URL on its own to measure its redirect chain. The checks that concern the whole host (the 404 test, the TLS certificate, the http:// and www versions, SPF) run once when the crawl starts and apply to every page, and a canonical is judged from the result of its target in the same crawl.

Limits and Politeness

The host gate allows one request in flight and at least one second between requests to the same host, for crawls and for every other QuikSEO request.

Site Rules a Crawl Checks

Besides the audit of every page, a crawl checks the site as a whole, judging only the pages it analysed: a page it skipped neither raises nor clears a finding.

Who Can Crawl a Website

Only the owner of a website can crawl it, after proving control of the site with a verification file.

1. Open Websites, choose the website, then Crawls.
2. Serve the file /quikseo-verification.txt with the line QuikSEO shows, then select Verify now.
3. Start the crawl; each audited page uses one audit of the monthly allowance, while redirects, errors and skipped URLs use none.

The same crawl can be started by an AI agent through the MCP tools start_crawl and get_crawl.

Related Documentation

• Every audit rule with its conditions and sources
• MCP server for AI coding agents
• Site contract for SEO conventions