tapki.com

TapkiBot

Open data collection robot tapki.com

TapkiBot downloads public pages of company websites. The collected information helps to update data about companies, their contacts and websites in the tapki.com service.

Token for robots.txt
TapkiBot
Information page
https://tapki.com/bot

Identification

When accessing a website, TapkiBot sends the following HTTP User-Agent header:

Mozilla/5.0 (compatible; TapkiBot/1.0; +https://tapki.com/bot)

Access control via robots.txt

TapkiBot complies with the Robots Exclusion Protocol. First he looks for rules for the group User-agent: TapkiBot. If there is no such group, the group rules apply User-agent: *The Allow and Disallow directives are supported.

Deny access to a specific section

For example, close the /private/ section.

User-agent: TapkiBot
Disallow: /private/

Deny access to the entire site

The ban applies to all paths on the site.

User-agent: TapkiBot
Disallow: /

If the server returns a 5xx error when requesting robots.txt, TapkiBot considers the entire site uncrawable.

How does TapkiBot work?

  • Before loading the page, it checks the robots.txt rules for the domain.
  • Uses HTTP GET requests and works without logging into an account.
  • Does not submit forms and only requests pages without authorization.
  • Transfers the permanent name of TapkiBot and the address of this page to the User-Agent.

Contacts

If you have questions about how TapkiBot works or want to report a problem, write to info@tapki.com.

Indicate the site domain and, if possible, the time of the request - this will help to quickly understand the request.