TapkiBot
Open data collection robot tapki.com
TapkiBot downloads public pages of company websites. The collected information helps to update data about companies, their contacts and websites in the tapki.com service.
- Token for robots.txt
- TapkiBot
- Information page
- https://tapki.com/bot
Identification
When accessing a website, TapkiBot sends the following HTTP User-Agent header:
Mozilla/5.0 (compatible; TapkiBot/1.0; +https://tapki.com/bot)Access control via robots.txt
TapkiBot complies with the Robots Exclusion Protocol.
First he looks for rules for the group User-agent: TapkiBot.
If there is no such group, the group rules apply User-agent: *The Allow and Disallow directives are supported.
Deny access to a specific section
For example, close the /private/ section.
User-agent: TapkiBot
Disallow: /private/Deny access to the entire site
The ban applies to all paths on the site.
User-agent: TapkiBot
Disallow: /If the server returns a 5xx error when requesting robots.txt, TapkiBot considers the entire site uncrawable.
How does TapkiBot work?
- Before loading the page, it checks the robots.txt rules for the domain.
- Uses HTTP GET requests and works without logging into an account.
- Does not submit forms and only requests pages without authorization.
- Transfers the permanent name of TapkiBot and the address of this page to the User-Agent.
Contacts
If you have questions about how TapkiBot works or want to report a problem, write to info@tapki.com.
Indicate the site domain and, if possible, the time of the request - this will help to quickly understand the request.