Advice Needed On Very Large Full Text Indexes

I'm currently building a product that will need to index and search roughly
500 million documents (perhaps more). Currently I have about 60 million
documents in the index (table size about 100GB and index size about 35GB)
Performance is (obviously) getting worse and I doubt my current architecture
will support the needed growth and still provide the required speed.

Documents are added 24x7 at the rate of 100's per minute (roughly 250,000
new ones each day). Plus, due to the search requirements, I had to clear
the noise file. I've been thinking of doing table partitioning (using
2005), creating indexed views and then using multiple full-text indexes to
query the data (however, of course, I will not be able to rank effectively
then).

So, I'm curious if any one else has this type of volume and has come up with
a solid solution. Appeciate any ideas.

I should also mention that I'm looking for short term consulting help on
this, as well as a full-time position - so while I'm grateful for free
suggestions from this board, should anyone be looking for work, please
contact me as well (company located in New Jersey).

Thanks!

Joel


.

Relevant Pages

  • Re: How to implement Lpars within Linux
    ... There are multiple choices in the HP world. ... logical partitioning of an HP server. ... So for example you can run Microsoft, Linux on ... The newer technology is supported on the their Integrity Servers. ...
    (linux.redhat)
  • Re: Newbie Qns: How do I... - Addendum
    ... >> Aragorn wrote: ... >>> As an example, here's my current partitioning layout, in the output ... That's a Mandrake 9.2 installation, ... root directory for this, i.e. */srv,* which can then have multiple ...
    (comp.os.linux.misc)
  • Re: Idiom for partitioning array?
    ... > Is there a nice Perl idiom for partitioning an array of length N ... (If N is not a multiple of n, ...
    (comp.lang.perl.misc)
  • Re: Idiom for partitioning array?
    ... > Is there a nice Perl idiom for partitioning an array of length N ... (If N is not a multiple of n, ...
    (comp.lang.perl.misc)
Loading