Google recently shared how SEOs and site owners can defend against malicious bots and enhance the performance of their websites.
Malicious Bots Are an SEO Problem
Many SEOs conducting site audits often overlook security and bot traffic because they aren’t aware that security issues can impact site performance.
Poor security can be a reason why a website is not being crawled effectively by Google. Even if you improve your core web vitals, a weak security posture could still drag down your site’s performance.
Every website faces attacks, and excessive bot crawling can trigger a “500 server error,” which prevents your site from serving web pages and hinders Google’s ability to crawl them.
How to Defend Against Bot Attacks
A user asked Google for advice on how to fight back against waves of scraper bots that were negatively impacting their server’s performance.
The user asked:
“Our website is experiencing significant disruptions due to targeted scraping by automated software, leading to performance issues, increased server load, and potential data security concerns. Despite IP blocking and other preventive measures, the problem persists. What can we do?”
Martin Splitt from Google suggested identifying the service responsible for the attacks and notifying them of the abusive use of their services. He also recommended using the firewall capabilities of a CDN (Content Delivery Network).
Martin responded:
“This sounds like a distributed denial-of-service issue if the crawling is so aggressive that it causes performance degradation. You can try identifying the owner of the network where the traffic is coming from, thank ‘their hoster’ and send an abuse notification. You can use WHOIS information for that, usually.
Alternatively, CDNs often have features to detect bot traffic and block it. By definition, they take the traffic away from your server and distribute it, so that’s a win. Most CDNs recognize legitimate search engine bots and won’t block them, but if that’s a concern, ask before you start using them.”
Will Google’s Advice Work?
Identifying the cloud provider or data center hosting the malicious bots is sound advice, but there are scenarios where this might not work.
Three Reasons Why Contacting Resource Providers Won’t Work:
Many Bots Are Hidden:
Bots often use VPNs and open-source “Tor” networks that mask their origin, making it impossible to identify the hosting infrastructure. Hackers also hide behind compromised computers, called botnets, to launch their attacks, making them untraceable.
Bots Switch IP Addresses:
Some bots evade IP blocking by quickly switching to a different network. An attack could start from a server in Germany and, when blocked, switch to a provider in Asia.
Inefficient Use of Time:
Contacting network providers about abusive users can be futile when the source of traffic is obscured or comes from numerous sources. Even acting against a small group of offenders can be inefficient since millions of bots will replace the ones you block.
The 80/20 Rule in Long-Tail Keywords
The term “long tail” comes from the shape of a distribution graph. If you plotted the popularity of keywords online, a few terms (like “Facebook,” “Malaysia tourism,” or “Durian festival”) would generate the majority of searches. However, these “head” terms account for only about 10-15% of total searches.
Interestingly, roughly 70% of web searches come from long-tail keywords. This means that while fewer people may be searching for specific terms like “organic durian farm near KL,” those who are searching for it are more likely to convert into customers.
Conclusion
Use a WAF to Block Bots
A Web Application Firewall (WAF) is a great option. Using a WAF like Skytomato Website Security, which routes traffic to a Website Security Firewall (WAF) will automatically blocks bots based on their behavior. For instance, if a bot is making excessive requests, it will block its IP. If the bot switches IP addresses, the WAF will detect the behavior and block it again.
At Skytomato we also provide Managed Website Security Enterprise where security are check against penetration test case deployed by corporations to test the security of the website.
We combine insightful and creative digital strategy with cutting-edge technology to deliver a full suite of web management services from infrastructure to website design and digital marketing strategies.
