Comments on: Shopping.com’s Spider Discovered http://comparisonengines.com/2006/10/03/shoppingcoms-spider-discovered/ Just another WordPress.com weblog Tue, 13 Mar 2012 02:47:07 +0000 hourly 1 http://wordpress.com/ By: Vic Berggren http://comparisonengines.com/2006/10/03/shoppingcoms-spider-discovered/#comment-744 Thu, 05 Oct 2006 00:07:42 +0000 http://www.comparisonengines.com/?p=559#comment-744 OK, they definitely crawled again today… roughly 1200 pages. Smart crawler too. It does not deviate too far from the product page at all and when it does its after the other category bits or a Data Sheet/Tech Spec for some of our Networking Products.

It’s also multi-threaded; grabbing between 8-10 pages per group.

]]> By: Vic Berggren http://comparisonengines.com/2006/10/03/shoppingcoms-spider-discovered/#comment-743 Wed, 04 Oct 2006 10:34:05 +0000 http://www.comparisonengines.com/?p=559#comment-743 Thanks for the mention(s) Brian!

I can analyze the crawl and see if a pattern emerges regarding frequency.

What’s interesting is that the site they crawled does not have an SDC PPC agreement. So, they were able to find the site (somehow) and locate the products.

Do you suspect that we may begin to see un_paid_for data at the bottom of a standard price comparison grid… sorta like ‘web results’?

]]> By: paulobrien http://comparisonengines.com/2006/10/03/shoppingcoms-spider-discovered/#comment-742 Wed, 04 Oct 2006 02:05:58 +0000 http://www.comparisonengines.com/?p=559#comment-742 So what is this saying?
How often will sdcresearchlabs-testbot access my web pages?
For most sites, sdcresearchlabs-testbot shouldn’t access your site more than once every few seconds on average.
They’ll crawl stores every few seconds… or not more than that…. so they aren’t answering the question…
If not daily, we’ll have problems

By the way, I just wrapped up my 3 part series about optimizing CSEs and summarizing some new CSE research. Check it out
http://seobrien.blogspot.com

]]>