Types of Bots

As explained in ‘How do bots work‘ – not all bots are the same. Let’s analyze what type of bots can BotMeNot use to test your website’s protection.

Free bot

A very, very basic bot

  • using very few IP addresses
  • using very few different User Agents
  • hitting very few URLs from your site multiple times
  • not trying to mimic any kind of human interaction on the visited page.

If you have any kind of anti-bot protection in place, a test using a Free bot should give good results.

Light bot

This type of bot will try some basic methods in order to stay undetected on your site

  • using a limited pool of IP addresses usingĀ a limited pool of different User Agents
  • hitting a limited number of URLs from your site multiple times
  • not trying to mimic any kind of human interaction on the visited page.

In our experience, Light bot should be able able to penetrate defenses of most off-the-shelf anti-bot solutions

Smart bot

This type of bot will try some advanced bot methods, which work well on 99% of websites that do employ anti-bot protection

  • using a large pool of IP addresses (very few requests will come from the same IP address)
  • using a large pool of different User Agents (so very few requests will get the same User-Agent)
  • hitting a number of URLs from your site multiple times – another method of staying ‘below the radar’
  • using some basic techniques to simulate human interaction on the visited page.
    – resolving standard Captcha’s

This kind of bot should be able to penetrate 99% of defenses. We use it in Price2Spy when we face a website with real good anti-bot protection.

Custom bot

A custom bot can do – literally anything, you name it. It’s developed by our engineers to simulate any kind of interaction on the page, for example

  • Logging in to a B2B website
  • Performing any sort of actions after successful login
  • (for an eCommerce site) Placing an order
  • Scrolling on and on in order to scrape content from pages with a lot of content (for example user reviews)
  • Resolving standard and non-standard Captcha’s
  • Persisting data from previous visits (in the form of cookies etc)
  • The size of the IP address pool is usually not that crucial with such bots (because a human visitor, who logs into a B2B site is not expected to use too many IP addresses)
  • Same counts for different User Agents

These types of bots are usually put in place for Social Networks, most popular tourist websites, and highly specialized websites that require users to log in before showing the data.