Re: Search Engine Indexing of Dynamic Sites

Tech Tip: Click here to run a free scan for Windows Errors and optimize PC performance

From: Jon Spivey (jons_at_mvps.org)
Date: 04/24/04 

Date: Sat, 24 Apr 2004 13:37:04 +0100

Tom,
The issue with this site is that each product has 5 similar products on its
product detail page on the home page there's 24 product links. So following
each of these links could eventually lead to every product in the catalogue
~300,000. When I've done similar sites with querystring links google doesn't
attempt to go that deep - seems to me that url rewriting has made google
follow many more links than they otherwise would, which is good - obviously
I want every product indexed.

Just need to do some more experimenting to stop the site crashing - can't
really blame my host for limiting sql server to 100 connections. In the
normal course of events that would be a reasonable limit. 

--
Cheers,
Jon
Microsoft MVP - FP
I've found with google that Thomas A. Rowe wrote:
> Jon,
>
> As you know I use Access, and have no major problems with Google, etc
> indexing the dynamic pages.
>
> Also most search engine seem to have no problem index the follow type
> of link:
>
> showproduct.aspx?productid=123
>
> whereas
>
> showproduct.aspx?productid=123&bin=124
>
> they will normal stop at the first querystring.
>
> But lately, Google has been following deeper.
>
>
> "Jon Spivey" <jons@mvps.org> wrote in message
> news:eh6MIVWKEHA.2784@TK2MSFTNGP10.phx.gbl...
>> Hi,
>>
>> I know this is  OT but thought it was interesting. As most of us
>> know when search engines see a dynamic page eg
>> showproduct.aspx?productid=123 they either won't follow the link
>> atall or will only follow a few.
>>
>> Yesterday I found a neat way to use asp.net to rewrite dynamic links
>> so that showproduct.aspx?productid=123 can be written as 123.aspx.
>> So far so good - the search engine doesnt see this as dynamic and
>> will happily follow the link. Tested it out on one of my sites and
>> seemed to work nicely. Then a few minutes ago google hit the site
>> followed all the links and opened 101 database connections in about
>> a minute - at which point the site fell over. This is against a sql
>> server 2000 db - obviously access would bottle out way before.
>>
>> Has anyone ever tried getting search engines to follow dynamic links
>> without that result?
>>
>> --
>> Cheers,
>> Jon
>> Microsoft MVP - FP

Relevant Pages

  • Risks Digest 24.70
    ... court case upended ... Search Engine Dispute Notifications: ... Extending Google Blacklists for Dispute Resolutions ...
    (comp.risks)
  • Google Agrees to Censor Results in China
    ... SAN FRANCISCO - Online search engine leader Google Inc. has agreed to ... Because of government barriers set up to suppress information, ... Google officials characterized the censorship concessions in China as ...
    (soc.culture.vietnamese)
  • The Google Goal Of Indexing
    ... The Google Goal Of Indexing ... Google's Goal of Quality Search ... Google started as a high quality search engine and continues to be the ... Information Retrieval System' filed by Google employee Anna Patterson ...
    (alt.internet.search-engines)
  • Re: Structuring informational content for commercial site
    ... >>>At giving a better rank to pages which have high valuable content. ... and I have sites/pages that rank well in Yahoo ... rolled out their new search engine - I ranked higher on them than I ... Google just took me a little while ...
    (alt.internet.search-engines)
  • errors saying that I have a malicious program
    ... When I am on msn.com and try to use their search engine, ... Are you trying to get to Google? ... download button while visiting a website. ... In Windows, open the Notepad program. ...
    (microsoft.public.windowsxp.security_admin)