I left Eurekster out since their focus is on a relevancy algorithm. When you have a full-on crawl (or use someone else’s) the billions of pages mean that you need a very good relevancy algorithm. Google has PageRank, at Become.com we use AIR technology to retrieve relevant results from our 3.2B indexed pages; Eurekster is looking at user feedback to get the best results.
For years at AltaVista and Yahoo! we looked at user feedback (specifically using CTR on web results) to improve relevancy, but there were always concerns about click fraud. The current social search guys have kinda a catch-22, if they become large and successful they create a huge incentive for spammers to try to add ‘bots to the community to skew the results. One way around this may be to have different social networks around each of the sites, so no one site has enough traffic to be worth trying to distort. This seems to be the promising approach that Eurekster is taking.

I chose to classify Fatlens as a scraper based upon the way that the system stores and presents information. Fatlens is extracting select, structured information from web pages so that users can do apples-to-apples comparisons of different ticket offerings. This is quite different than the full page indexing that a general search engine like Google does.
To be clear, Fatlens does do a web crawl (http://fatlens.com/main/fatbot.php) to find the tickets pages it extracts data from (most scrapers do some form of web crawling). If you could provide any additional insight into how FatBot operates (# of sites crawled, how the sites are selected, etc.) that would be very insightful.

- How accurate and comprehensive does the data need to be?
- What technical (human) resources are available? If you have access to 60+ PhD’s you’re obviously in a different position than if you have only a B.S. in C.S. fresh out of college.
- How many sites will need to be crawled/scraped/extracted?
- What’s the time-line? Sophisticated crawlers can take significantly longer than straightforward template-based extraction engines.

I actually wrote a blog entry not too long ago entitled “Three common methods for data extraction” that may be helpful to some: http://blog.screen-scraper.com/2006/03/21/three-common-methods-for-data-extraction/. It’s a bit more general, and not necessarily targeted to comparison engines. It does, however, also delineate advantages and disadvantages to various approaches.

Thanks for taking the time to help educate the comparison engines audience on Vertical Web Searches. Thank you also for mentioning FatLens in your post. However, I’d like to point out that your assumption about our technology is incorrect. FatLens is not an example of a scraper.

According to your classification scheme, we would best be described as a crawler aimed at creating a vertical search solution for shopping.

Siva Kumar

PS. The post was very illustrative of how one would view text or Web page search technologies. Shopping as a vertical search area could use a bit more discussion on alternative technology approaches.