ToolvyDownloader

ToolvyDownloader is the crawler behind Toolvy’s Website Downloader. It fetches pages only when a person asks it to save a copy of a site, never on its own schedule, and it does not index anything. It sends this User-Agent:

Mozilla/5.0 (compatible; ToolvyDownloader/1.0; +https://toolvy.xyz/bot/)

How it behaves

  • Follows robots.txt: a group for ToolvyDownloader if you have one, otherwise the * group, including Crawl-delay.
  • Waits between requests to the same host, and backs off when a site answers 429 or 503 (it honours Retry-After).
  • Stops after at most 1,000 pages or 45 minutes per job.
  • Skips DRM-protected and live video, and does not get past paywalls or logins.

Blocking it

To keep it off your whole site, add this to your robots.txt:

User-agent: ToolvyDownloader
Disallow: /

Or only keep it out of part of the site:

User-agent: ToolvyDownloader
Disallow: /private/
Crawl-delay: 5

A person who downloads their own site can choose to ignore robots.txt; the tool asks them to do that only for sites they own or have permission to copy. To report misuse, see DMCA & takedown.