To provide you with results, Qwant maintains a continuously updated index of public web pages. Here’s how it works.
Crawlers
Qwant uses crawlers (robots d'exploration) that automatically browse the web. QwantBot — Qwant’s main crawler — is identifiable in your server logs.
The process happens in two steps:
Crawling
- The crawler discovers new pages by following links from already known pages or via a sitemap file.
- It respects the directives in each domain’s robots.txt file.
- It downloads the HTML content of accessible pages (HTTP code 200, without blocking by captcha or error 403).
Indexing
- Each crawled page is evaluated according to quality criteria.
- Pages that pass this evaluation are indexed and become candidates for display in the results.
Crawl Frequency
The frequency with which our crawlers visit a site depends on:
- The freshness of the content: a regularly updated site is visited more often.
- The popularity of the site: sites with many incoming links are prioritized.
- The performance of the server: if your site is slow, we adjust our pace to avoid overloading it.
- The directives in your robots.txt.
What Is Indexed
Qwant indexes publicly accessible HTML pages, as well as several file formats: PDF, office documents (Word, Excel, PowerPoint, OpenDocument), plain text.
What Is Not Indexed
- Pages protected by a password or an authentication system.
- Pages blocked by robots.txt (
Disallow: /for QwantBot). - Content requiring complex client-side rendering or user interaction.
- Pages marked noindex via the meta tag:
<meta name="robots" content="noindex">.
Tips for Publishers
To optimize your site’s indexing by Qwant:
- Publish an up-to-date sitemap.xml file (standard, news, or video).
- Maintain a clear robots.txt that does not block QwantBot.
- Carefully structure the HTML of your pages.
- Publish quality, up-to-date, and well-structured content.