QwantBot is Qwant's official crawler. It crawls the web to build and refresh the index that powers our search engine.
This page explains how to identify it in your logs, how to verify that a request really comes from Qwant, and how to control its access to your site.
Identifying QwantBot in your logs
Depending on the bot version, the User-Agent varies slightly. One element remains constant and makes it possible to identify it: the keyword Qwantbot, as a single word.
The general format is as follows:
Mozilla/5.0 (compatible; Qwantbot{-news}/X.Y_{worker_id}; +https://help.qwant.com/bot/)The elements inside braces { } are optional: -news identifies the bot dedicated to Qwant News, and worker_id identifies the machine that initiated the request.
Here are two examples of User-Agents with which we may visit your site:
Mozilla/5.0 (compatible; Qwantbot/1.0_12345; +https://help.qwant.com/bot/) Mozilla/5.0 (compatible; Qwantbot-news/2.0; +https://help.qwant.com/bot/)
Verifying that a request really comes from Qwant
A User-Agent is purely declarative: any bot can claim to be QwantBot. If you have doubts about requests observed in your logs, two methods can settle the matter.
Recommended method — reverse DNS lookup
- Perform a reverse DNS lookup on the IP address that initiated the request, and verify that it points to a name ending in
qwant.com. - Additionally, perform a forward DNS lookup on this name to confirm that it points back to the same IP address.
On Linux, using the host command:
> host 91.242.162.1 1.162.242.91.in-addr.arpa domain name pointer qwantbot-1-162-242-91.qwant.com. > host qwantbot-1-162-242-91.qwant.com qwantbot-1-162-242-91.qwant.com has address 91.242.162.1
If the name obtained does not end in qwant.com, the request does not come from our servers.
Alternative method — IP address ranges
You can also compare the request's IP address with the list of our ranges published in this file: qwantbot.json
Controlling QwantBot via robots.txt
QwantBot follows the robot exclusion standard described on robotstxt.org
The robots.txt file is placed at the root of each relevant subdomain: a file present on example.com does not apply to blog.example.com.
Allow QwantBot throughout the site — this is the default behavior if no directive targets it:
User-agent: Qwantbot Allow: /
Disallow crawling of a directory:
User-agent: Qwantbot Disallow: /prive/
Disallow crawling of the entire site:
User-agent: Qwantbot Disallow: /
Space out visits with Crawl-delay
If QwantBot makes requests to your server too frequently, the Crawl-delay directive sets a minimum delay, in seconds, between two requests:
User-agent: Qwantbot Crawl-delay: 8
robots.txt or a noindex tag?
The two mechanisms address different needs and are not interchangeable.
| Mechanism | Effect | Use it to |
|---|---|---|
robots.txt |
The page is not crawled, and therefore not downloaded. It cannot be indexed. | Reduce the load on your server, exclude entire sections of the site |
Tag <meta name="robots" content="noindex">
|
The page is downloaded but does not appear in search results. | Remove a specific page from the results while keeping it accessible |
Visit frequency
QwantBot adjusts its crawling frequency based on the size and popularity of your site. You can influence it in two ways:
- by publishing a valid and up-to-date sitemap.xml;
- by maintaining good server performance, as a fast site is crawled more often.
Report unusual behavior
If something went wrong while our bots were crawling your site, we are sincerely sorry.
Report any problem to us by email at qwantbot@qwant.com, specifying:
- your domain name;
- a sample of logs containing the problematic requests;
- the period concerned.