A new report has been released by Distil Networks, Inc. and Clareity Consulting with a focus on web scraping, a term many real estate agents might not yet be familiar with.
The report, “2015 Study on Real Estate Website Scraping and MLS Data Security,” examines two surveys that sought to understand the barriers to adoption of anti-scraping technology—one with Multiple Listing Service (MLS) executives, and another with the owners of companies that provide Internet Data Exchange (IDX) websites and Virtual Office Websites (VOW).
“The real estate industry faces serious challenges with intellectual property infringement via web scraping, yet they’ve been slow to adopt effective anti-scraping tools,” says Rami Essays, co-founder and CEO of Distil Networks. “Scrapers use bots to grab online real estate content, like property listings, and then create derivative products such as lead gen programs and appraisal data.”
The study reveals the perceptions of MLS and IDX executives regarding web scraping and how each member of the MLS data supply chain can align to protect the integrity of real estate industry data.
The Findings
MLS Executives – One hundred MLS executives responded to the survey, representing over half a million MLS members, brokers, and agents.
· 99 percent of MLS executives say compliance with rules protecting misuse of MLS data is important and 95 percent agree that IDX sites should be subject to rules specifically mandating anti-scraping protections.
· 98 percent of respondents want a set of standardized tests to facilitate anti-scraping compliance reviews, yet no test currently exists.
· 94 percent of MLS executives indicate a vendor’s information security practices and sophistication of anti-scraping technology are important to them when selecting a technology vendor.
IDX/VOW Vendors – 14 vendors representing over 400,000 websites were surveyed. IDX and VOW vendors operate on behalf of the MLS to serve agent sites and other websites that display and syndicate MLS listing data.
· Prior to this study, 43 percent of IDX/VOW vendors surveyed were not aware of how pervasive bots and scraping were across the entire industry.
· 62 percent rated compliance with MLS rules as the most important factor in having IDX/VOW vendors implement an anti-scraping solution.
· Only 17 percent of IDX/VOW vendors are using a commercial service to protect against scraping. Some are using services with limited protection, especially against sophisticated web scraping bots.
“Most IDX/VOW vendors are using reactive monitoring mechanisms and other outdated methods to stop scrapers, which are not effective against today’s nefarious actors. For those attempting to implement proactive tools to combat scraping, almost all indicated they were using old IP-address-based methods, like rate limiting, CAPTCHAs, web application firewalls and IP blocking services. On their own, these methods are not effective,” says Matt Cohen for Clareity Consulting. “Based on the near unanimous support for rule changes and a standard for testing compliance, combined with advances in affordable anti-scraper services available, there is a path forward for the industry to solve the scraping problem once and for all.”
To download a copy of the full report, visit 2015 Study: State of Web Scraping Data Theft Across Real Estate Websites & MLS Data.
