Re: Exchange 2003 Private Store Corruption

:-(

Yeah - look at the articles I posted in my last reply. -1018's are never
fun. I think you're on the right track with creating a new mailbox store.

Once you've pulled the faulty RAM, create the new mailbox store and move
everyone that you can to that new store. I'd recommend that on the first
pass, leave the defaults - this will make it so that only the users with
problems will end up being skipped. Once you've moved everyone else, then
you can start working on the problem mailboxes.

Once you have everyone else moved, dismount the problem mailbox store, and
create a copy of it. Then, remount it, and do the move mailbox again, this
time telling it to skip corrupt items. You may have to try the move a few
times, bumping up the amount of corrupt items it can encounter.

Depending on how much data is lost, you can try to either restore from
backup to the Recovery Storage Group, and recover lost mail that way, or you
can use something like Quest's Recovery manager to do the same thing.

You could also at that point look at using eseutil against the copy to try
and fix it, then mount it to the Recovery Storage Group.

--
Ben Winzenz
Exchange MVP
MessageOne
Read my blog!
http://winzenz.blogspot.com
http://feeds.feedburner.com/winzenz (RSS Feed)


"Corbin O'Reilly" <corbinoreilly@xxxxxxxxxxxxx> wrote in message
news:F2dtg.8317$IF2.1593@xxxxxxxxxxxxxxxxxxxxxxxxx
Hi Ben. Here is the error I am getting. Please advise. Thanks.
Information Store (2856) First Storage Group: The database page read from
the file "E:\NewMailboxStore\NewMailbox Store (GTMAIL).edb" at offset
5251796992 (0x0000000139081000) (database page 1282176 (0x139080)) for
4096 (0x00001000) bytes failed verification due to a page checksum
mismatch. The expected checksum was 1873244556488189379
(0x19ff19ffcee9a5c3) and the actual checksum was 5178919806808169513
(0x47df38200d091c29). The read operation will fail with error -1018
(0xfffffc06). If this condition persists then please restore the database
from a previous backup. This problem is likely due to faulty hardware.
Please contact your hardware vendor for further assistance diagnosing the
problem.

For more information, click http://www.microsoft.com/contentredirect.asp.

For more information, see Help and Support Center at


"Ben Winzenz [Exchange MVP]" <ben_winzenz@nospamdotmessageonedotcom> wrote
in message news:u$BCCMdpGHA.524@xxxxxxxxxxxxxxxxxxxxxxx
It could be that it is just DIMM 4 that is the problem. Do you have to
have DIMM's installed in pairs in this server? If not, I'd just pull out
DIMM 4.

Also, can you post the entire contents of the error event? Does it
mention something to the effect of

Information Store (<PID>) The database page read from the file "<path to
edb file>" at offset <offset> for <value> bytes failed verification due
to a page checksum mismatch. The expected checksum was <checksum> and the
actual checksum was <checksum>. The read operation will fail with error
<error code> (<error code>). If this condition persists then please
restore the database from a previous backup.

and is the error code referenced a -1018 error? If so, then that *is* a
database corruption issue, not just a corrupt mailbox. Even if that is
the case, I still wouldn't recommend using eseutil or isinteg - those are
really last resort utilities. Moving mailboxes is one option. Restoring
from backup should be another option you should look at, or restoring
from backup to the Recovery Storage Group.

You might also look at the storage group properties and see if you have
the option "Zero out deleted database pages" option checked, and if so,
you might consider unchecking it.

Also, do you have any Exchange 2003 SP's installed? If not, once you get
this fixed, you should really make sure you are up to date. SP1 includes
an additional error correcting code that is designed to fix many of
the -1018 errors that are as a result of a flipped bit.

Here are some KB articles that talk about 474 errors and -1018 errors.
http://support.microsoft.com/default.aspx?scid=kb;en-us;810411
http://support.microsoft.com/kb/327334/
http://support.microsoft.com/default.aspx?scid=kb;en-us;867626
http://support.microsoft.com/default.aspx?scid=kb;en-us;314917
http://support.microsoft.com/default.aspx?scid=kb;en-us;812531 - webcast
on understanding and resolving -1018 errors.

In your case, it really would appear that faulty memory is the underlying
cause.

--
Ben Winzenz
Exchange MVP
MessageOne
Read my blog!
http://winzenz.blogspot.com
http://feeds.feedburner.com/winzenz (RSS Feed)


"Corbin O'Reilly" <corbinoreilly@xxxxxxxxxxxxx> wrote in message
news:uj9tg.89246$qd2.10626@xxxxxxxxxxxxxxxxxxxxxxxxx
Thanks for the info Ben. Yes they are mounted now so hopefully you are
right and they will mount properly after the reboot. The HP Insight
Online Diagnostic Utility says "Correctable Memory Threshold Exceeded"
for DIMM 4 so it does look like the memory is faulty. Have you seen 474
errors before? In your experience what leads to a 474 error? The
description points to hardware. I am exporting mailboxes to PST files
right now. Did you see Mitch's previous response. I am just trying to
get the server back up and running without the faulty memory installed.
My plan is to create a new store, move as many uncorrupted mailboxes as
I can to it, and then try to repair the old stores.

"Ben Winzenz [Exchange MVP]" <ben_winzenz@nospamdotmessageonedotcom>
wrote in message news:%23iIUsWcpGHA.3600@xxxxxxxxxxxxxxxxxxxxxxx
Inline.

--
Ben Winzenz
Exchange MVP
MessageOne
Read my blog!
http://winzenz.blogspot.com
http://feeds.feedburner.com/winzenz (RSS Feed)


"Corbin O'Reilly" <corbinoreilly@xxxxxxxxxxxxx> wrote in message
news:OX7tg.8250$IF2.4659@xxxxxxxxxxxxxxxxxxxxxxxxx
Hello everyone. I need some help. We have an Exchange 2003 Enterprise
Edition Server running Windows 2003 Enterprise Edition R2 server. The
machine is a HP DL380 G4 server with 2 GB of memory and has been
online for about two months. The only thing running on this server is
Exchange 2003 and Trend-Micro AntiVirus for Exchange. The server came
standard with 2 - 512 MB HP-brand DIMMs. We purchased additional
memory from CDW. They are 2 -512 MB EDGE-brand DIMMs and are supposed
to be compatible. On the first day that we powered up the server the
server said there was an issue with DIMM 4, the EDGE-brand chip. After
a reboot the error went away. About three weeks ago I noticed that the
server had not completed its online backup properly and two of our
employees were having Calendar issues. I checked the Event Viewer and
it had tons of 474 errors which says that there is probably a hardware
problem. I created a new store on the server and moved all mailboxes
to the new store. The two mailboxes with the corrupted Calendars would
not move so I left them in the corrupted store until I got back from
vacation. While on vacation for two weeks I checked the NTBACKUP logs
daily and the online backups were completing properly. Everything was
fine for almost the entire time I was gone. The day before I got back
from vacation I got calls from 5 employees saying they were having
strange Exchange problems. Some were not getting new e-mails (we use
cached mode), others had Inboxes that were empty, etc. I checked the
NTBACKUP log and sure enough the new store I created is now corrupted
too with a 474 error. My theory is that the EDGE-brand memory chip is
acting up and corrupting the store. What I would like to do is
shutdown the server and remove the 2 - 512 MB EDGE-brand memory DIMMs
from server, and boot it back up. I have a couple of questions:

1. The server is partitioned into a RAID 1 OS C-Drive and a RAID 6
Exchange Data E-Drive. The server currently has 2 GB of memory and the
page file is located on the C-Drive and is set to Initial
: 3072 and Maximum: 6144. Since I will be removing the 2 - 512 MB
EDGE-brand DIMMs the new total memory size will be 1024 MB (1GB).
Should I adjust the page file to Initial: 1536 and Maximum 3072 in
Windows 2003 BEFORE or AFTER I remove the memory?

I don't think it much matters whether you do it before or after,
though, except that if you do it afterwards, you will have to reboot a
second time.


2. Will the server perform OK with 1 GB of memory for a while? We have
about 75 mailboxes and a few public folders.

More than likely, it will be just fine. It doesn't sound like your
server is heavily loaded


3. Since both Private Info stores are corrupted will they mount
properly when the server is rebooted?

Are they mounted right now? If so, then once you reboot, they should
be able to mount. Dealing with corrupt mailboxes is somewhat different
than dealing with an actual corrupt database. Corrupt databases are
not mountable. Databases with corrupt mailboxes usually are mountable,
but the mailbox itself may or not be able to be fixed. I'd consider
using Exmerge to export whatever you can from the corrupt mailboxes.
If they can be moved, great. If not, then the mailboxes should be
deleted and re-create on the new store.


I am trying desperately to avoid being dead in the water tonight. My
concern is that the stores will not come up and we will be totally
down. I am getting more complaints from different employees so the
problem seems to be spreading and getting worse. I would appreciate
your expertise on this problem. Thanks.

As an additional note, have you run any server diagnostics to test the
new memory? What is your plan if removing the memory doesn't curtail
the problem? It's really better to have a true diagnosis that the
memory is faulty before you pull it out. HP should have some
diagnostics that you can run to test the memory. I'd do that first
before you pull it out.












.

Relevant Pages

  • Re: Exchange 2003 Private Store Corruption
    ... Once you've pulled the faulty RAM, create the new mailbox store and move ... then you can start working on the problem mailboxes. ... have DIMM's installed in pairs in this server? ... Online Diagnostic Utility says "Correctable Memory Threshold Exceeded" ...
    (microsoft.public.exchange.admin)
  • Re: no mailboxs in private foulder
    ... limited experince with exchange. ... Ping the exchange server ... Right click the mailbox store, is the first option "mount store" or is it ... Where are you seeing "Mailboxes and stuff"? ...
    (microsoft.public.exchange.admin)
  • Re: public folders are not all in sync
    ... Exchange 2003 server about the calendar connector. ... then just dismount the public store. ... Network Administrator mailboxes are supposed to be moved. ... I don't have very many folders so I ...
    (microsoft.public.exchange.admin)
  • Re: Exchange 2003 Private Store Corruption
    ... Information Store First Storage Group: The database page read from ... Online Diagnostic Utility says "Correctable Memory Threshold Exceeded" ... the server back up and running without the faulty memory installed. ...
    (microsoft.public.exchange.admin)
  • Re: Exchange 2003 Private Store Corruption
    ... Diagnostic Utility says "Correctable Memory Threshold Exceeded" for DIMM 4 ... Edition Server running Windows 2003 Enterprise Edition R2 server. ... I created a new store on ... Dealing with corrupt mailboxes is somewhat different than ...
    (microsoft.public.exchange.admin)