IncrediChatBot

What it is

IncrediChat gives website owners an AI chat assistant that answers visitors' questions from the owner's own content. To build that content, an IncrediChat customer can ask us to read pages of a website: typically their own, or one they are authorised to use. IncrediChatBot is the program that fetches those pages.

It is not a search engine crawler. It does not index the web, it does not follow links to other websites, and it only runs when a customer has set up a website source in IncrediChat. Pages are used only to answer questions in that customer's chat assistant.

How to recognise it

Every request ends its User-Agent header with this token:

IncrediChatBot/1.0 (+https://incredichat.com/bot)

The rest of the header looks like an ordinary browser, because many sites serve different or broken pages to unknown agents. Match on IncrediChatBot in your logs or firewall rules.

How it behaves

• Polite. About one request per second to a site, one site at a time per customer. It honours Crawl-delay in robots.txt, up to 10 seconds.
• Follows robots.txt. It reads /robots.txt first and obeys the rules for IncrediChatBot (or * if there is none). The only exception is a customer who tells us they own the site; in that case they accept responsibility for its rules.
• Reads, never writes. It only sends GET requests. It does not log in, fill in forms, run JavaScript, or call your APIs.
• Stays on your site. Same website only, at most 5 redirects, HTML pages only, 5 MB per page.
• Skips the obvious. By default it skips login, cart, checkout, account, admin, search and feed paths, and files such as images and PDFs. Customers can narrow it further to specific sections.
• Backs off. If your server returns errors or blocks it, it stops and retries later, with longer gaps each time, and eventually pauses itself. A block never causes the customer's existing content to be deleted.
• Re-checks occasionally. Customers can ask for daily or weekly updates, usually run overnight in their time zone.

How to block it

Add this to your /robots.txt to keep it out of the whole site:

User-agent: IncrediChatBot
Disallow: /

Or only part of it:

User-agent: IncrediChatBot
Disallow: /members/
Crawl-delay: 5

You can also block the user-agent token at your firewall or CDN. If you do not want a particular IncrediChat customer to use your content, tell us and we will contact them and stop the crawls.