Balancing the Sharing of Information

CyberSecurity Journal

Subscribe to CyberSecurity Journal: eMailAlertsEmail Alerts newslettersWeekly Newsletters
Get CyberSecurity Journal: homepageHomepage mobileMobile rssRSS facebookFacebook twitterTwitter linkedinLinkedIn


Security Authors: Shelly Palmer, Slavik Markovich, Elizabeth White, Greg Ness, Liz McMillan

Related Topics: Security Journal, CyberSecurity Journal

Blog Post

Screen Scraping Bots By @DISTIL | @CloudExpo [#Cloud]

Thieves and unauthorized content aggregators use screen scraping bots and cheap labor to steal your content

Four Ways Screen Scraping Bots Degrade Your Website's Success

Like DDoS attacks, aggressive screen scraping can create huge spikes in traffic, cause brownouts, and even take you down for extended periods of time. Web scraping doesn't make the headlines and most people aren't aware of the damage it can cause to your website's success.

Thieves and unauthorized content aggregators use screen scraping bots and cheap labor to steal your content, repackage it for sale, or place it on unauthorized sites - all of which hurt your SEO, negatively impact your brand, and reduce your ability to monetize your intellectual property. Below I've outlined just a few of the many ways that screen scraping can impact a company and its website.

Screen scraping bots can:

  1. Steal your intellectual property: Your pricing data, product information, web page copy, and more is all vulnerable to web scraping. Once screen scrapers get their hands on it, they either use it to their competitive advantage, post it to another site, or sell it off. In any case, it means your proprietary data falls into the hands of others - for free. This malicious activity can be illustrated when, for example, an ecommerce company runs promotions on popular products and then, minutes later, sees their competitors undercutting their own pricing to get ahead.
  2. Bog down your web infrastructure: Screen scraping consumes bandwidth and compute resources, forcing IT teams to over-provision infrastructure to account for random spikes in bot traffic. Massive screen scraping requests can slow down your website or take you down for extended periods of time. This can happen when bots aggressively screen scrape long tail listings from a directory website, for example. These types of listings do not usually get traffic, so they are not cached in the company's CDN (Content Delivery Network). When bots aggressively scrape these listings, it causes a brown out and user complaints.
  3. Harm your reputation: Brownouts, outages, and duplicate content can harm your reputation. Screen scrapers can repost your content on less reputable sites as well as introduce errors into your content. Sometimes, screen scraping bots register on websites in order to scrape content behind a registration form. Fake registrations can pollute your site and back-end systems and have negative cascading effects. Social media sites often have a lot of fake registrations, which translate into fake social media profiles. When such sites send out monthly newsletters, for example, to fake emails resulting from fake registrations, they incur substantial bounce back rates. This hurts their email reputation, impacting their overall email delivery rates, which leads to lost opportunities to engage prospects and customers, and ultimately lost revenue.
  4. Put your SEO performance at risk: If your stolen content shows up elsewhere on the web or page loading times slow down significantly, it can have a huge impact on your site's search engine rankings. In some cases, it can get you blacklisted altogether, essentially rendering your brand invisible on the web. When Google changed its algorithm to penalize duplicate content, many publishers lost great portions of their traffic, some up to 70%, resulting in near bankruptcy and severe financial turmoil.

Screen scraping is a serious threat that can affect your company's site, customers, and sales in an instant. Websites depend on unique content, fast response times, and security to maintain brand integrity and a competitive edge. Want to protect yourself and your business? Recognize the threats that screen scraping can impose and establish an action plan to outsmart malicious intruders.

More Stories By Rami Essaid

Rami Essaid is the Co-founder and CEO of Distil Networks, the global leader in bot detection and mitigation. He began his career as the founder and CEO of Chit Chat Communications. After a successful exit, he consulted in mobile development.

With over 11 years in communications, network security, and infrastructure management, Rami advised enterprise companies to help improve scalability and reliability while maintaining a high level of security. He attended North Carolina State University where he studied computer engineering.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.