ToolvyDownloader
ToolvyDownloader is the crawler behind Toolvy’s Website Downloader. It fetches pages only when a person asks it to save a copy of a site, never on its own schedule, and it does not index anything. It sends this User-Agent:
Mozilla/5.0 (compatible; ToolvyDownloader/1.0; +https://toolvy.xyz/bot/)
How it behaves
- Follows robots.txt: a group for
ToolvyDownloaderif you have one, otherwise the*group, includingCrawl-delay. - Waits between requests to the same host, and backs off when a site answers 429 or 503 (it honours
Retry-After). - Stops after at most 1,000 pages or 45 minutes per job.
- Skips DRM-protected and live video, and does not get past paywalls or logins.
Blocking it
To keep it off your whole site, add this to your robots.txt:
User-agent: ToolvyDownloader Disallow: /
Or only keep it out of part of the site:
User-agent: ToolvyDownloader Disallow: /private/ Crawl-delay: 5
A person who downloads their own site can choose to ignore robots.txt; the tool asks them to do that only for sites they own or have permission to copy. To report misuse, see DMCA & takedown.