There are two perspectives (at least) in every web scraping story: one of the website being scraped and the other of the scraper itself. In this article, we’ll talk about this relationship using a specific example. On one side, there’s Bax Shop, a website selling and renting musical equipment. On the other, there’s Price2Spy – an advanced web crawler, looking to monitor prices on Bax Shop.
We’ll start off by talking about how Bax Shop came to be and give you a glimpse into its roots. Then we’ll focus on the period when they first launched their online store and how initially they fought bots on their website. Afterward, we’ll go over other methods they employed and also cover how Price2Spy adapted to these defenses, in this back-and-forth game of price monitoring.
Read on!
Introduction – Bax Brothers story – how did Bax Shop come to be?
From their very beginnings, up until this moment, one thing that has remained constant for Bax Brothers is their passion for music. From when the older brother made his first disco ball (only at the age of 10), up until now — when Bax Shop has a 280 000 square foot warehouse with over 48 000 products — love for music has been the driving force behind their success. Let’s go into a bit more detail into how that in-between period looked like and what nudged them into launching an online store.
The business side of the story starts when Bax Brothers decided to start DJing. For this endeavor, they had to buy expensive equipment. They did so, and it paid off relatively soon, since they were good at DJing. But soon, they figured out that there’s no reason for the equipment to gather dust while they aren’t DJing, so they decided to rent it out. They did so for the first time in 1998.
Their next step, however, illustrates perfectly well how important it is to be a pioneer in a certain field. Namely, in 2005 they started renting out and selling equipment online. In hindsight, it seems that it was this step that kickstarted the vigorous growth of Bax Shop. After that, and up until now, it has been mostly a history of steady growth over the years.
Chapter One – Initial Bot Protection Challenges
In order to penetrate the music equipment market, Bax Shop decided to start off by offering products at extremely low prices. They started to seriously implement this strategy around 2013/2014, and up until then, the EU musical market hadn’t experienced a similar disruption. Germany, Netherlands, and France were their first markets, but slowly but surely they expanded. Today, besides these three countries, they are doing business in Belgium, Spain, Italy, Sweden, and the United Kingdom.
The emergence of Bax Shop coincided with price monitoring as a new standard in the musical industry. Price2Spy, a very advanced price monitoring tool, already had clients from this industry, and Bax Shop was immediately put on the radar. This means that existing sellers of musical equipment recognized Bax Shop as a significant competitor and decided to monitor their prices.
However, due to its founder’s background, Bax Shop was also one of the pioneers in bot detection. Price2Spy noticed this in fall 2014 when unsuccessful price checks occurred from time to time, with an increasing frequency. The default solution to this problem is to introduce a new IP address to the price monitoring network. However, this didn’t really work out. To be more specific, what happened is that barely minutes after introducing a new IP address to the Price2Spy network, it got blocked by Bax Shop.
Price2Spy’s solution to this problem was to introduce completely new technology into the game. Stealth IP traffic is an advanced price monitoring approach that makes it essentially impossible for monitored websites to distinguish between human and bot traffic. To make things a bit more clear – those websites could try to block unwanted traffic, but the risk of blocking out genuine human traffic is too high. Furthermore, Bax Shop was the first website where Price2Spy used this new technology.
Chapter Two – New Approaches, New Solutions
As years went by, Price2Spy noticed that Bax shop was changing the way they deal with bots. In 2018 they have loosened up the anti-bot rules. It’s safe to assume that due to their strictness, those rules were actually blocking out genuine human traffic. A new approach was needed.
Bax reverted to other methods of fighting bots—increasingly frequent changes to the format of pages on the website. What most likely Bax Shop wasn’t aware of is that the team at Price2Spy was already accustomed to this type of defense. This means that in some sense, it was a regression from their previous solution.
To make it a bit more clear – changes to the format of the website’s pages are done in order to prevent automated scraping. Simply put, the logic behind this defense is to make the crawler read either wrong or non-existing information, in a place on a page where there once was real information. It’s an approach that has very little effect on humans. Most human visitors are accustomed to small changes on websites or don’t even notice them.
This can, however, present a problem for web crawlers, the basic ones that is. But, as we’ve mentioned, it was an easy obstacle for Price2Spy to overcome. This is why Bax Shop had to double think whether reverting to this approach was a good decision.
Chapter Three – Same Old, Same Old
It seems that Bax Brothers came to a conclusion that going back to stricter rules regarding bot protection was needed. Once again, they’ve started banning certain IP addresses. However, this meant only one thing. Stealth IP traffic was turned back on for Bax Shop.
It’s important to mention that even though Stealth IP traffic is a powerful weapon against these types of defenses, it does come with one caveat. It’s a very expensive approach. It requires a much more advanced infrastructure compared to regular price monitoring.
This all means that currently, Bax Shop is a difficult, but not impossible, website to monitor. It does require advanced techniques, but an advanced web scraper, such as Price2Spy is able to penetrate all of its defenses.
Conclusion – It’s a game of cat and mouse
Today, Bax Shop, in every European country, where they opened operations, is enjoying quite a success. They are no longer considered a newcomer to the music equipment industry, but rather an established market player, with a really high reputation. This also means that they don’t need to resort to aggressive pricing policies, as they did in the beginning, such as penetration pricing.
All together, Bax Shop is a good example of an IT-aware company that is aware of business harm that can be done by bots. Also, they went (and are still going) a long way to prevent that harm. On the other hand, Price2Spy is also a good example of a highly adaptable and versatile web scraper.
Now, when all is said and done, we can say that two crucial factors that will determine success in this field are innovation and adaptability. It’s up to websites such as Bax Shop to come up with innovative measures against bots, and up to web scrapers such as Price2Spy to find ways to penetrate those defenses. There is no rest in the world of web scraping.