Re: Search Engine Indexing of Dynamic Sites

From: Thomas A. Rowe (tarowe_at_mvps.org)
Date: 04/24/04 

Date: Sat, 24 Apr 2004 09:27:22 -0400

Jon,

Are you keeping the project data connection to the database open at all times, and then just opening
and closing your individual table/recordset(s) as needed?

Based on what you have indicated, you have no more than 2 queries/recordsets/connections on your
product detail page, which shouldn't be a problem.

I have two sites where Google has indexed 8,600 and 14,000 pages respectively, so far this month.

Originally Google crashed the site with 8,600 pages when then sent 5 spiders which generated 500+
queries per second. At that time I had 3+26 queries on the listing page which checked to see which
Letter of the Alphabet had listings as well as the displaying the listings for the current selected
letter.

If you are paying for MS SQL server and any extra bandwidth, I can not see why a host would limit
the number of connections. 

-- 
==============================================
Thomas A. Rowe (Microsoft MVP - FrontPage)
WEBMASTER Resources(tm)
http://www.ycoln-resources.com
FrontPage Resources, WebCircle, MS KB Quick Links, etc.
==============================================
To assist you in getting the best answers for FrontPage support see:
http://www.net-sites.com/sitebuilder/newsgroups.asp
"Jon Spivey" <jons@mvps.org> wrote in message news:etPjajfKEHA.1312@TK2MSFTNGP12.phx.gbl...
> Tom,
> The issue with this site is that each product has 5 similar products on its
> product detail page on the home page there's 24 product links. So following
> each of these links could eventually lead to every product in the catalogue
> ~300,000. When I've done similar sites with querystring links google doesn't
> attempt to go that deep - seems to me that url rewriting has made google
> follow many more links than they otherwise would, which is good - obviously
> I want every product indexed.
>
> Just need to do some more experimenting to stop the site crashing - can't
> really blame my host for limiting sql server to 100 connections. In the
> normal course of events that would be a reasonable limit.
>
> --
> Cheers,
> Jon
> Microsoft MVP - FP
>
> I've found with google that Thomas A. Rowe wrote:
> > Jon,
> >
> > As you know I use Access, and have no major problems with Google, etc
> > indexing the dynamic pages.
> >
> > Also most search engine seem to have no problem index the follow type
> > of link:
> >
> > showproduct.aspx?productid=123
> >
> > whereas
> >
> > showproduct.aspx?productid=123&bin=124
> >
> > they will normal stop at the first querystring.
> >
> > But lately, Google has been following deeper.
> >
> >
> > "Jon Spivey" <jons@mvps.org> wrote in message
> > news:eh6MIVWKEHA.2784@TK2MSFTNGP10.phx.gbl...
> >> Hi,
> >>
> >> I know this is  OT but thought it was interesting. As most of us
> >> know when search engines see a dynamic page eg
> >> showproduct.aspx?productid=123 they either won't follow the link
> >> atall or will only follow a few.
> >>
> >> Yesterday I found a neat way to use asp.net to rewrite dynamic links
> >> so that showproduct.aspx?productid=123 can be written as 123.aspx.
> >> So far so good - the search engine doesnt see this as dynamic and
> >> will happily follow the link. Tested it out on one of my sites and
> >> seemed to work nicely. Then a few minutes ago google hit the site
> >> followed all the links and opened 101 database connections in about
> >> a minute - at which point the site fell over. This is against a sql
> >> server 2000 db - obviously access would bottle out way before.
> >>
> >> Has anyone ever tried getting search engines to follow dynamic links
> >> without that result?
> >>
> >> --
> >> Cheers,
> >> Jon
> >> Microsoft MVP - FP
>
>

Relevant Pages

  • Re: Search Engine Indexing of Dynamic Sites
    ... When I've done similar sites with querystring links google doesn't ... > As you know I use Access, and have no major problems with Google, etc ... > Also most search engine seem to have no problem index the follow type ... > "Jon Spivey" wrote in message ...
    (microsoft.public.frontpage.client)
  • Risks Digest 24.70
    ... court case upended ... Search Engine Dispute Notifications: ... Extending Google Blacklists for Dispute Resolutions ...
    (comp.risks)
  • Google Agrees to Censor Results in China
    ... SAN FRANCISCO - Online search engine leader Google Inc. has agreed to ... Because of government barriers set up to suppress information, ... Google officials characterized the censorship concessions in China as ...
    (soc.culture.vietnamese)
  • The Google Goal Of Indexing
    ... The Google Goal Of Indexing ... Google's Goal of Quality Search ... Google started as a high quality search engine and continues to be the ... Information Retrieval System' filed by Google employee Anna Patterson ...
    (alt.internet.search-engines)
  • Re: Structuring informational content for commercial site
    ... >>>At giving a better rank to pages which have high valuable content. ... and I have sites/pages that rank well in Yahoo ... rolled out their new search engine - I ranked higher on them than I ... Google just took me a little while ...
    (alt.internet.search-engines)