JET/ESE failure when backing up Exchange 2003 Online Stores

Hello,

Operating System: Windows 2003 SBS
Service Packs: Windows 2003 SP1, Exchange 6.5 SP1, Sharepoint SP1, SBS SP1.

Prelim: When performing an online backup, the job fails, logging the
following information.
=============================
NTBackup Log:
-----------------------------
Backup started on 30/07/2005 at 8:41 PM.
The 'Microsoft Information Store' returned 'Error returned from an ESE
function call (d).

' from a call to 'HrESEBackupRead()' additional data '-'The 'Microsoft
Information Store' returned 'Error returned from an ESE function call (d).

' from a call to 'HrESEBackupRead()' additional data '-'
The operation was ended.
Backup completed on 30/07/2005 at 8:42 PM.
Directories: 0
Files: 1
Bytes: 4,391,610
Time: 15 seconds
=============================
Event Log
-----------------------------
Information Store (4372) First Storage Group: The database page read from
the file "D:\Exchange\MDBDATA\priv1.edb" at offset 4435968
(0x000000000043b000) for 4096 (0x00001000) bytes failed verification due to a
page checksum mismatch. The expected checksum was 843791235
(0x00000000324b3b83) and the actual checksum was 4647998405507
(0x0000043a324b3b83). The read operation will fail with error -1018
(0xfffffc06). If this condition persists then please...
-----------------------------
Information Store (4372) First Storage Group: Error (-1018) during backup of
a database (file D:\Exchange\MDBDATA\priv1.edb). The database will be unable
to restore.
-----------------------------
Information Store (4372) First Storage Group: Ending the backup of the file
D:\Exchange\MDBDATA\priv1.edb. Not all data in the file has been read (read
4390912 bytes out of 5592723456 bytes).
==============================

Here's what we've done so far...

- VolumeShadowCopy patches (not applicable to SBS SP1).
- Offline eseutil defrag, checksum and integrity.
- Offline isinteg running all tests on both stores (see below).

==============================
Isinteg logs (two)
-----------------------------
C:\Program Files\Exchsrvr\bin>isinteg -s fs1 -fix -test alltests
Databases for server fs1:
Only databases marked as Offline can be checked

Index Status Database-Name
Storage Group Name: First Storage Group
1 Offline Mailbox Store (FS1)
2 Offline Public Folder Store (FS1)
Enter a number to select a database or press Return to exit.
1
You have selected First Storage Group / Mailbox Store (FS1).
Continue?(Y/N)y
Test Categorization Tables result: 0 error(s); 0 warning(s); 0 fix(es); 0
row(s)
; time: 0h:0m:3s
Test Restriction Tables result: 0 error(s); 0 warning(s); 0 fix(es); 0
row(s); t
ime: 0h:0m:9s
Test Search Folder Links result: 0 error(s); 0 warning(s); 0 fix(es); 0
row(s);
time: 0h:0m:0s
Test Global result: 0 error(s); 0 warning(s); 0 fix(es); 1 row(s); time:
0h:0m:0
s
Test Delivered To result: 0 error(s); 0 warning(s); 0 fix(es); 1 row(s);
time: 0
h:0m:0s
Test Repl Schedule result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s);
time:
0h:0m:0s
Test Timed Events result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s);
time: 0
h:0m:0s
Test reference table construction result: 0 error(s); 0 warning(s); 0
fix(es); 0
row(s); time: 0h:2m:10s
Test Folder result: 1 error(s); 5 warning(s); 3 fix(es); 1260 row(s); time:
0h:1
m:41s
Test Deleted Messages result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s);
tim
e: 0h:0m:0s
Test Message result: 0 error(s); 0 warning(s); 0 fix(es); 81471 row(s);
time: 0h
:5m:24s
Test Attachment result: 0 error(s); 0 warning(s); 0 fix(es); 37261 row(s);
time:
0h:0m:1s
Test Mailbox result: 0 error(s); 0 warning(s); 0 fix(es); 25 row(s); time:
0h:0m
:0s
Test Sites result: 0 error(s); 0 warning(s); 0 fix(es); 95 row(s); time:
0h:0m:0
s
Test Categories result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s); time:
0h:
0m:0s
Test Per-User Read result: 0 error(s); 0 warning(s); 0 fix(es); 1070 row(s);
tim
e: 0h:0m:0s
Test special folders result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s);
time
: 0h:0m:0s
Test Message Tombstone result: 0 error(s); 0 warning(s); 0 fix(es); 59
row(s); t
ime: 0h:0m:0s
Test Folder Tombstone result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s);
tim
e: 0h:0m:0s
Test reference count verification result: 0 error(s); 0 warning(s); 4
fix(es); 0
row(s); time: 0h:0m:2s
Now in test 21(Row Count/Dumpster Count) of total 21 tests; 100% complete.


C:\Program Files\Exchsrvr\bin>isinteg -s fs1 -fix -test alltests
Databases for server fs1:
Only databases marked as Offline can be checked

Index Status Database-Name
Storage Group Name: First Storage Group
1 Offline Mailbox Store (FS1)
2 Offline Public Folder Store (FS1)
Enter a number to select a database or press Return to exit.
2
You have selected First Storage Group / Public Folder Store (FS1).
Continue?(Y/N)y
Test Categorization Tables result: 0 error(s); 0 warning(s); 0 fix(es); 0
row(s)
; time: 0h:0m:0s
Test Restriction Tables result: 0 error(s); 0 warning(s); 0 fix(es); 0
row(s); t
ime: 0h:0m:5s
Test Search Folder Links result: 0 error(s); 0 warning(s); 0 fix(es); 0
row(s);
time: 0h:0m:1s
Test Global result: 0 error(s); 0 warning(s); 0 fix(es); 1 row(s); time:
0h:0m:0
s
Test Delivered To result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s);
time: 0
h:0m:0s
Test Repl Schedule result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s);
time:
0h:0m:0s
Test Timed Events result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s);
time: 0
h:0m:0s
Test reference table construction result: 0 error(s); 0 warning(s); 0
fix(es); 0
row(s); time: 0h:0m:27s
Test Folder result: 0 error(s); 0 warning(s); 7361 fix(es); 637 row(s);
time: 0h
:1m:2s
Test Deleted Messages result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s);
tim
e: 0h:0m:0s
Test Message result: 0 error(s); 0 warning(s); 0 fix(es); 22612 row(s);
time: 0h
:1m:2s
Test Attachment result: 0 error(s); 0 warning(s); 0 fix(es); 13202 row(s);
time:
0h:0m:0s
Test Per-User Read result: 0 error(s); 0 warning(s); 0 fix(es); 262 row(s);
time
: 0h:0m:0s
Test Replication result: 0 error(s); 0 warning(s); 0 fix(es); 4 row(s);
time: 0h
:0m:0s
Test Message Tombstone result: 0 error(s); 0 warning(s); 0 fix(es); 3
row(s); ti
me: 0h:0m:0s
Test Folder Tombstone result: 0 error(s); 0 warning(s); 0 fix(es); 8 row(s);
tim
e: 0h:0m:0s
Test reference count verification result: 0 error(s); 0 warning(s); 0
fix(es); 0
row(s); time: 0h:0m:0s
Now in test 18(Row Count/Dumpster Count) of total 18 tests; 100% complete.


C:\Program Files\Exchsrvr\bin>isinteg -s fs1 -fix -test alltests
Databases for server fs1:
Only databases marked as Offline can be checked

Index Status Database-Name
Storage Group Name: First Storage Group
1 Offline Mailbox Store (FS1)
2 Offline Public Folder Store (FS1)
Enter a number to select a database or press Return to exit.
1
You have selected First Storage Group / Mailbox Store (FS1).
Continue?(Y/N)y
Test Categorization Tables result: 0 error(s); 0 warning(s); 0 fix(es); 0
row(s)
; time: 0h:0m:0s
Test Restriction Tables result: 0 error(s); 0 warning(s); 0 fix(es); 0
row(s); t
ime: 0h:0m:0s
Test Search Folder Links result: 0 error(s); 0 warning(s); 0 fix(es); 0
row(s);
time: 0h:0m:2s
Test Global result: 0 error(s); 0 warning(s); 0 fix(es); 1 row(s); time:
0h:0m:0
s
Test Delivered To result: 0 error(s); 0 warning(s); 0 fix(es); 1 row(s);
time: 0
h:0m:0s
Test Repl Schedule result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s);
time:
0h:0m:0s
Test Timed Events result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s);
time: 0
h:0m:0s
Test reference table construction result: 0 error(s); 0 warning(s); 0
fix(es); 0
row(s); time: 0h:2m:7s
Test Folder result: 0 error(s); 5 warning(s); 0 fix(es); 1260 row(s); time:
0h:1
m:58s
Test Deleted Messages result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s);
tim
e: 0h:0m:0s
Test Message result: 0 error(s); 0 warning(s); 0 fix(es); 81471 row(s);
time: 0h
:5m:33s
Test Attachment result: 0 error(s); 0 warning(s); 0 fix(es); 37261 row(s);
time:
0h:0m:1s
Test Mailbox result: 0 error(s); 0 warning(s); 0 fix(es); 25 row(s); time:
0h:0m
:0s
Test Sites result: 0 error(s); 0 warning(s); 0 fix(es); 95 row(s); time:
0h:0m:0
s
Test Categories result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s); time:
0h:
0m:0s
Test Per-User Read result: 0 error(s); 0 warning(s); 0 fix(es); 1070 row(s);
tim
e: 0h:0m:0s
Test special folders result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s);
time
: 0h:0m:0s
Test Message Tombstone result: 0 error(s); 0 warning(s); 0 fix(es); 59
row(s); t
ime: 0h:0m:0s
Test Folder Tombstone result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s);
tim
e: 0h:0m:0s
Now in test 20(reference count verification) of total 20 tests; 100%
complete.

C:\Program Files\Exchsrvr\bin>isinteg -s fs1 -fix -test alltests
Databases for server fs1:
Only databases marked as Offline can be checked

Index Status Database-Name
Storage Group Name: First Storage Group
1 Offline Mailbox Store (FS1)
2 Offline Public Folder Store (FS1)
Enter a number to select a database or press Return to exit.
2
You have selected First Storage Group / Public Folder Store (FS1).
Continue?(Y/N)y
Test Categorization Tables result: 0 error(s); 0 warning(s); 0 fix(es); 0
row(s)
; time: 0h:0m:0s
Test Restriction Tables result: 0 error(s); 0 warning(s); 0 fix(es); 0
row(s); t
ime: 0h:0m:0s
Test Search Folder Links result: 0 error(s); 0 warning(s); 0 fix(es); 0
row(s);
time: 0h:0m:1s
Test Global result: 0 error(s); 0 warning(s); 0 fix(es); 1 row(s); time:
0h:0m:0
s
Test Delivered To result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s);
time: 0
h:0m:0s
Test Repl Schedule result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s);
time:
0h:0m:0s
Test Timed Events result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s);
time: 0
h:0m:0s
Test reference table construction result: 0 error(s); 0 warning(s); 0
fix(es); 0
row(s); time: 0h:0m:27s
Test Folder result: 0 error(s); 0 warning(s); 0 fix(es); 638 row(s); time:
0h:0m
:37s
Test Deleted Messages result: 0 error(s); 0 warning(s); 0 fix(es); 0 row(s);
tim
e: 0h:0m:0s
Test Message result: 0 error(s); 0 warning(s); 0 fix(es); 22612 row(s);
time: 0h
:1m:1s
Test Attachment result: 0 error(s); 0 warning(s); 0 fix(es); 13202 row(s);
time:
0h:0m:0s
Test Per-User Read result: 0 error(s); 0 warning(s); 0 fix(es); 262 row(s);
time
: 0h:0m:0s
Test Replication result: 0 error(s); 0 warning(s); 0 fix(es); 4 row(s);
time: 0h
:0m:0s
Test Message Tombstone result: 0 error(s); 0 warning(s); 0 fix(es); 3
row(s); ti
me: 0h:0m:0s
Test Folder Tombstone result: 0 error(s); 0 warning(s); 0 fix(es); 8 row(s);
tim
e: 0h:0m:0s
Now in test 17(reference count verification) of total 17 tests; 100%
complete.
==============================

We're currently being forced to do offline backups which is less than ideal.
We receive similar errors in 3rd party backup apps.

Any help will be greatly appreciated,

Adam Risbey

.

Relevant Pages

  • Re: JET/ESE failure when backing up Exchange 2003 Online Stores
    ... I was going to suggest that you create a new mailbox store and move all the ... > - Offline isinteg running all tests on both stores. ... > Only databases marked as Offline can be checked ... > 1 Offline Mailbox Store (FS1) ...
    (microsoft.public.exchange.admin)
  • Re: Portable stored procedures
    ... > JDatastore lets you store binary streams outside the table scema. ... Other databases usually do not store BLOB's ... > you will have to write the function in the client in Java (or whatever ... >> stored procedures written in some proprietary language. ...
    (comp.lang.java.databases)
  • Re: Creating Many Databases In A Large Store to Do Defrags
    ... Thank you-this will help instead of the Defrag only some of the mailboxes are ... I would probably try to keep each store below 100GB. ... you should never need to do an offline defrag. ... databases it will help but I have tried to explain that if a whole bunch ...
    (microsoft.public.exchange.admin)
  • Re: Creating Many Databases In A Large Store to Do Defrags
    ... I mean sure, you can keep splitting them up and splitting them up, that's a good idea. ... Just move all the mailboxes in that store somewhere else, ... Gig, one has the 362 I mentioned, one has 195 and one has ... databases it will help but I have tried to explain that if a whole bunch of ...
    (microsoft.public.exchange.admin)
  • Re: Online, Offline, Disconnected applications.
    ... I would recommend using SQLite as client-side database. ... supporting both databases). ... (although I don't think database synchronization is part of the offering). ... > and the Offline Application Block for for guidance. ...
    (microsoft.public.dotnet.framework.adonet)