Qwant Web crawler

Introduction

Qwant uses web crawlers to enhance its index and provide the best possible service. This page gives information about how they work and their behaviour on your websites.

User-agent

During crawlings, we announce ourselves with different user-agents depending on the version of our crawler. Something that will always appear in our user-agents that you can use to identify our web crawlers is the name: Qwantify or Qwant-news.

Here are a few examples of user-agents we might crawl you with:

Mozilla/5.0 (compatible; Qwantify/2.4w; +https://www.qwant.com/)
Mozilla/5.0 (compatible; Qwant-news/2.0; +https://www.qwant.com/)
Mozilla/5.0 (compatible; Qwantify/Bleriot/1.1; +https://help.qwant.com/bot)

Robots.txt

The crawler respects the robots rules standard described at http://www.robotstxt.org/wc/exclusion.html#robotstxt

Verifying Crawler

To check if a web crawler accessing your server is from Qwant:

  • Make a DNS lookup on the IP address

  • Check that domain name contains “search.qwant.com

  • Make a forward lookup on the previous domain name

  • Check if IP address is the same as the original one

Examples:

> host 91.242.162.87
87.162.242.91.in-addr.arpa domain name pointer qwantbot-91-242-162-87.search.qwant.com.
> host qwantbot-91-242-162-87.search.qwant.com
qwantbot-91-242-162-87.search.qwant.com has address 91.242.162.87

Troubles

If something went wrong when we visited your website using our crawlers, we are sincerely sorry for the inconvenience.

Please report us any problem caused by the crawlers by connecting to this url: https://about.qwant.com/fr/contact/