Re: Can you use web search engines as front ends to your database?

From: Nick Malik [Microsoft] (nickmalik_at_hotmail.nospam.com)
Date: 01/29/05 

Date: Sat, 29 Jan 2005 09:57:16 -0800

Search engines crawl each web site, often months after they "discover" their
existence (which they do when you tell them about you, by listing on one of
the listing services). The Microsoft Small Business Center has a nice
registration service that informs search engines of the existence of your
site (see http://www.microsoft.com/smallbusiness/bc/default.mspx )

So, if you want a search engine to return a hit when a part number is
entered, you need to make sure that that:

a) your web site has a complete, navigable, tree containing your entire
catalog, with part numbers and complete descriptions of the product on every
page. Crawlers will not run your Search function, nor will they run
Javascript, nor will they follow a link that varies from another link only
by a parameter, so it has to be straight-out-navigation. This usually
requires you to implement an HTTPModule
(http://www.devx.com/vb2themax/Article/19901) so that you can embed the
parameters directly into the name of the "resource".

For example, browse over to Amazon. Drill down from the menus. You will
find that, technically, you can get to EVERY product in the Amazon catalog
without using Javascript and by simply linking from one page to another.
After you've found a book, look at the URL. It will be a long custom string
(they invented their own page engine called obidos). Select all parts of
the string BEFORE the ?. (That is the only part that a crawler will keep).
Now, copy that into another browser window. You will find the exact same
book. In other words, the crawler needs to find the part by urls that look
like this:
   http://site.example.com/catalog/part42134321
and not a url that looks like this
   http://site.example.com/showpage.aspx?part=42134321

b) you should find a way to increase your link visibility, by getting as
many other sites to link to yours as possible. Crawlers find you that way
too. I recommend that you look at partnerships and relationships with other
sites, so that they will link you frequently. Once again, the links should
be as specific as possible and must not rely on parameters to find the part.

c) pray that your part numbers are relatively unique compared to part
numbers from other products or from other companies.

Good Luck, 

-- 
--- Nick Malik [Microsoft]
    MCSD, CFPS, Certified Scrummaster
    http://blogs.msdn.com/nickmalik
Disclaimer: Opinions expressed in this forum are my own, and not
representative of my employer.
   I do not answer questions on behalf of my employer.  I'm just a
programmer helping programmers.
--
<disaia@earthlink.net> wrote in message
news:1106954641.684742.137190@z14g2000cwz.googlegroups.com...
> 2 problems:
>
> Example: If a person types in a part number into Yahoo:
>
> 1. Is there a way for Yahoo to list your web site as one of the
> results.
>
> 2. If the user clicks on your link, can your web application know the
> part number the user typed into the Yahoo search box.  I would like to
> use that part number to query our database and present a dynamic web
> page with the details of that part number.
>
>
> I know one way to solve this is to create a web page for every part
> number, but since there are over 10 million part numbers it wouldn't be
> the best solution.  I need a way to dynamically generate that web page
> (hopefully with VB.NET, my prefered language).
> Thanks,
> Dominic Isaia
> disaia@earthlink.net
>

Relevant Pages

  • Re: The Proverbial Cat online store now open
    ... >> and visit it, and link to it, and search engines will index you accordingly ... then linking to it increases it's ranking in google. ... >> free with all the major crawlers listed on this page. ... >> follow links, so if you have good links pointing at your Web site, ...
    (rec.arts.sf.tv.babylon5.moderated)
  • Professional Web Site Submission!
    ... Would you like your web site to be Found and Seen on the Major search engines? ... You could very well be losing clients that would have booked your service had they found you on the search engines. ... Perhaps they submitted it to a handful of search engines but to gain any worthwhile results from the internet you need to be actively and aggressively submitted by a company that specializes in exactly this. ... We believe that our service represents the best and most effective web site submission package available on the internet today. ...
    (comp.unix.sco.misc)
  • Professional Web Site Submission!
    ... Would you like your web site to be Found and Seen on the Major search engines? ... You could very well be losing clients that would have booked your service had they found you on the search engines. ... Perhaps they submitted it to a handful of search engines but to gain any worthwhile results from the internet you need to be actively and aggressively submitted by a company that specializes in exactly this. ... We believe that our service represents the best and most effective web site submission package available on the internet today. ...
    (comp.unix.sco.misc)
  • Re: The Proverbial Cat online store now open
    ... > and visit it, and link to it, and search engines will index you accordingly ... then linking to it increases it's ranking in google. ... > follow links, so if you have good links pointing at your Web site, ... > you have a Web site, build a link to your CafePress.com Shop. ...
    (rec.arts.sf.tv.babylon5.moderated)
  • the Major Search Engines! We can change that!
    ... Your web site designer and your host are NOT responsible for getting your web site seen! ... they may submit your site to a few search engines periodically but to stay on the TOP of the search engines is an aggressive business. ... We guarantee that your web site will be FOUND and SEEN! ... Guaranteed Submission the Major Search Engines. ...
    (comp.unix.sco.misc)
Loading