Types of Bots

As explained in ‘How do bots work‘ – not all bots are the same. Let’s analyze what type of bots can BotMeNot use to test your website’s protection.

Starter bot

A very, very basic bot

  • using very few IP addresses
  • using very few different User Agents
  • hitting very few URLs from your site multiple times
  • not trying to mimic any kind of human interaction on the visited page.

If you have any kind of bot protection in place, a test using a Smart bot should give good results.

Light bot

This type of bot will try some basic methods in order to stay undetected on your site

  • using a limited pool of IP addresses using a limited pool of different User Agents
  • hitting a limited number of URLs from your site multiple times
  • not trying to mimic any kind of human interaction on the visited page.

In our experience, Light bot should be able able to penetrate defenses of most off-the-shelf bot solutions

Smart bot

This type of bot will try some advanced bot methods, which work well on 99% of websites that do employ bot protection

  • using a large pool of IP addresses (very few requests will come from the same IP address)
  • using a large pool of different User Agents (so very few requests will get the same User-Agent)
  • hitting a number of URLs from your site multiple times – another method of staying ‘below the radar’
  • using some basic techniques to simulate human interaction on the visited page.
    – resolving standard Captcha’s

This kind of bot should be able to penetrate 99% of defenses. We use it in Price2Spy when we face a website with real good bot protection.

Custom bot

A custom bot can do – literally anything, you name it. It’s developed by our engineers to simulate any kind of interaction on the page, for example

  • Logging in to a B2B website
  • Performing any sort of actions after successful login
  • (for an eCommerce site) Placing an order
  • Scrolling on and on in order to scrape content from pages with a lot of content (for example user reviews)
  • Resolving standard and non-standard Captcha’s
  • Persisting data from previous visits (in the form of cookies etc)
  • The size of the IP address pool is usually not that crucial with such bots (because a human visitor, who logs into a B2B site is not expected to use too many IP addresses)
  • Same counts for different User Agents

These types of bots are usually put in place for Social Networks, most popular tourist websites, and highly specialized websites that require users to log in before showing the data.

API bot

Unlike regular websites, which are meant to be read by humans, API services are supposed to be consumed by applications/automated processes. Therefore, there are some specifics when it comes to API bot protection testing.

  • API service expects to have an automated process as its consumer,
  • API service consists of endpoints (instead of webpages) – some publicly listed, some not,
  • Overloading API servers is very common and a real risk,
  • Implementing request rates (request throttling) is the go-to solution against overload,
  • API bot protection test will let you know if the request throttling and your other protection methods are working properly