Discussion:
failed drives, rebuilds, exchange corruptions, oh my!
(too old to reply)
Joe SoandSo
2009-06-10 05:26:27 UTC
Permalink
Ok, So I am looking for a little direction and possibly some ideas on how to
handle a problem that has happened on an sbs2k server. First a little
history. The server has a raid 5 drive configuration. Last week one of the
three drives failed. I promptly bought a replacement drive and put it into
the system. About 5 percent into the rebuild the rebuild failed, the
replacement drive went to a failed state, and another of the drives also
failed. So I had two drives in the failed state! I called Dell (it is a 6
year old dell server), they said that they have seen when older drives
rebuild they sometimes fail because of a timeout issue that is fixed in
newer firmware for the harddrives. So, long story short, we forced the
newly failed drive back on, we used a server2k3 recovery console to run
chkdsk, this made the server again bootable (it would blue screen during
bootup before we ran chkdsk), then we updated the harddrive firmware on all
three harddrives and attempt another rebuild. This rebuild worked and
everything appeared very good. However I am now noticing errors in the
application log about exchange. I will attach the errors to the end of this
message (sorry for the long post). Some of the errors happen during
bootup. I've tried rebooting a few times but the same errors occur. Other
errors happen at night. I've only seen this one night, tonight will be the
next night to see if it happens again. Backupexec also gave an error.

What do you think I am up against? And what do you suggest I should I do.
My last good backup was Friday Night the 5th.

Now as I see it I have several options:
.Restore the whole system from backup .. con's - I hope it works well, what
do I do with all the changes/new files/new email since Friday.
.Restore just exchange - con's - how do I preserve changes made since
Friday? Will this really fix it.
.Call Microsoft support and try to repair the existing database - will there
be more problems/missing things down the road.

On top of all this, I would like to somehow snapshot the entire system on to
a spare harddrive(s) so that if something goes wrong in the recovery
procedure I can go back to the current state. Does anyone have any
recommendation on how to do this? Mirror? Swap/in/out a few drives into the
raid? Image software to usb drive?

Your suggestions are greatly appreciated. I will check back tomorrow to see
what suggestions you might have.

Thanks Again!

error messages:

======================================================================

Media Name: "Media created 7/20/2004 09:31:32 AM"
Backup of "\\NETSERVER\Microsoft Exchange Mailboxes "
Backup set #1 on storage media #1
Backup set description: "Nightly Full Backup"
Backup Type: FULL - Back Up Files - Reset Archive Bit
Backup started on 6/8/2009 at 9:00:29 PM.

Backup completed on 6/8/2009 at 11:40:33 PM.
Backed up 116973 mail message(s) in 1602 folder(s) in 37 mailbox(es)
Processed 9,926,872,461 bytes in 2 hours, 40 minutes, and 4 seconds.
Throughput rate: 59.1 MB/min
----------------------------------------------------------------------

Media Name: "Media created 7/20/2004 09:31:32 AM"
Backup of "\\NETSERVER\Microsoft Information Store\First Storage Group "
Backup set #2 on storage media #1
Backup set description: "Nightly Full Backup"
Backup Type: FULL - Database & Logs (flush committed logs)
Backup started on 6/8/2009 at 11:40:35 PM.
Access denied to database .
^ ^ ^ ^ ^
WARNING: "Mailbox Store (NETSERVER)" is a corrupt file.
This file cannot verify.
^ ^ ^ ^ ^
Log files

Backup completed on 6/9/2009 at 12:54:33 AM.
Backed up 2 Exchange Server store(s)
Backed up Exchange Server logs
1 corrupt file was backed up
Processed 16,759,227,384 bytes in 1 hour, 13 minutes, and 58 seconds.
Throughput rate: 216.1 MB/min
----------------------------------------------------------------------


Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 302
Date: 6/9/2009
Time: 5:04:14 PM
User: N/A
Computer: NETSERVER
Description:
Information Store (4584) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: The database
engine has successfully completed recovery steps.

For more information, click http://www.microsoft.com/contentredirect.asp.


Event Type: Information
Event Source: MSExchangeIS Public Store
Event Category: General
Event ID: 1216
Date: 6/9/2009
Time: 5:04:21 PM
User: N/A
Computer: NETSERVER
Description:
The information store database "First Storage Group\Public Folder Store
(NETSERVER)" is limited to 16384 MB.

For more information, click http://www.microsoft.com/contentredirect.asp.

Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 301
Date: 6/9/2009
Time: 5:04:08 PM
User: N/A
Computer: NETSERVER
Description:
Information Store (4584) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: The database
engine has begun replaying logfile E:\Program
Files\Exchsrvr\mdbdata\E00.log.

For more information, click http://www.microsoft.com/contentredirect.asp.

Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 301
Date: 6/9/2009
Time: 5:04:00 PM
User: N/A
Computer: NETSERVER
Description:
Information Store (4584) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: The database
engine has begun replaying logfile E:\Program
Files\Exchsrvr\mdbdata\E000E656.log.

For more information, click http://www.microsoft.com/contentredirect.asp.

Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 300
Date: 6/9/2009
Time: 5:02:41 PM
User: N/A
Computer: NETSERVER
Description:
Information Store (4584) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: The database
engine is initiating recovery steps.

For more information, click http://www.microsoft.com/contentredirect.asp.

Event Type: Information
Event Source: ESE
Event Category: General
Event ID: 102
Date: 6/9/2009
Time: 5:02:35 PM
User: N/A
Computer: NETSERVER
Description:
Information Store (4584) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: The database
engine started a new instance (0).

For more information, click http://www.microsoft.com/contentredirect.asp.

Event Type: Warning
Event Source: Perflib
Event Category: None
Event ID: 1016
Date: 6/9/2009
Time: 5:02:33 PM
User: N/A
Computer: NETSERVER
Description:
The data buffer created for the "EXOLEDB" service in the "C:\Program
Files\Exchsrvr\bin\exodbpc.dll" library is not aligned on an 8-byte
boundary. This may cause problems for applications that are trying to read
the performance data buffer. Contact the manufacturer of this library or
service to have this problem corrected or to get a newer version of this
library.
Data:
0000: 0c 19 8a 00 04 15 00 00 ..Š.....

Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 213
Date: 6/9/2009
Time: 12:54:34 AM
User: N/A
Computer: NETSERVER
Description:
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: The backup
procedure has been successfully completed.

For more information, click http://www.microsoft.com/contentredirect.asp.

Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 225
Date: 6/9/2009
Time: 12:54:34 AM
User: N/A
Computer: NETSERVER
Description:
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: No log files
can be truncated.

For more information, click http://www.microsoft.com/contentredirect.asp.

Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 223
Date: 6/9/2009
Time: 12:54:17 AM
User: N/A
Computer: NETSERVER
Description:
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: Starting the
backup of log files (range E:\Program Files\Exchsrvr\mdbdata\E000E63B.log -
E:\Program Files\Exchsrvr\mdbdata\E000E63E.log).

For more information, click http://www.microsoft.com/contentredirect.asp.

Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 221
Date: 6/9/2009
Time: 12:54:17 AM
User: N/A
Computer: NETSERVER
Description:
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: Ending the
backup of the file E:\Program Files\Exchsrvr\mdbdata\pub1.stm.

For more information, click http://www.microsoft.com/contentredirect.asp.

Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 222
Date: 6/9/2009
Time: 12:41:43 AM
User: N/A
Computer: NETSERVER
Description:
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: Ending the
backup of the file E:\Program Files\Exchsrvr\mdbdata\priv1.edb. Not all data
in the file has been read (read 11242700800 bytes out of 16215977984 bytes).

For more information, click http://www.microsoft.com/contentredirect.asp.

Event Type: Error
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 217
Date: 6/9/2009
Time: 12:41:43 AM
User: N/A
Computer: NETSERVER
Description:
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: Error (-1018)
during backup of a database (file E:\Program
Files\Exchsrvr\mdbdata\priv1.edb). The database will be unable to restore.

For more information, click http://www.microsoft.com/contentredirect.asp.

Event Type: Error
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 474
Date: 6/9/2009
Time: 12:41:43 AM
User: N/A
Computer: NETSERVER
Description:
Information Store (4628) The database page read from the file "E:\Program
Files\Exchsrvr\mdbdata\priv1.edb" at offset 11242729472 (0x000000029e1e7000)
for 4096 (0x00001000) bytes failed verification due to a page checksum
mismatch. The expected checksum was 4205941953 (0xfab194c1) and the actual
checksum was 571156525 (0x220b282d). The read operation will fail with
error -1018 (0xfffffc06). If this condition persists then please restore
the database from a previous backup.

For more information, click http://www.microsoft.com/contentredirect.asp.

Event Type: Error
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 474
Date: 6/9/2009
Time: 12:41:41 AM
User: N/A
Computer: NETSERVER
Description:
Information Store (4628) The database page read from the file "E:\Program
Files\Exchsrvr\mdbdata\priv1.edb" at offset 11242729472 (0x000000029e1e7000)
for 4096 (0x00001000) bytes failed verification due to a page checksum
mismatch. The expected checksum was 4205941953 (0xfab194c1) and the actual
checksum was 571156525 (0x220b282d). The read operation will fail with
error -1018 (0xfffffc06). If this condition persists then please restore
the database from a previous backup.

For more information, click http://www.microsoft.com/contentredirect.asp.

Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 220
Date: 6/8/2009
Time: 11:40:47 PM
User: N/A
Computer: NETSERVER
Description:
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: Begining the
backup of the file E:\Program Files\Exchsrvr\mdbdata\priv1.edb (size 15464
Mb).

For more information, click http://www.microsoft.com/contentredirect.asp.
Merv Porter [SBS-MVP]
2009-06-10 11:53:01 UTC
Permalink
It looks like you've hit the 16GB database limit for your Exchange store.
You may be able to temporarily increase the limit to 17GB but you'll need to

How to temporarily increase the Exchange 2000 16-gigabyte database size
limit
http://support.microsoft.com/kb/813051

Exceeding the 16GB Limit in Exchange 2000/2003
http://thelazyadmin.com/blogs/thelazyadmin/archive/2005/06/22/Exceeding-the-16GB-Limit-in-Exchange-2000_2F00_2003.aspx

Exchange Standard and the 16GB Limit
http://hellomate.typepad.com/exchange/2004/03/exchange_standa.html

It may be time to consider upgrading to SBS 2008 (or at least SBS 2003).
--
Merv Porter [SBS-MVP]
============================
Post by Joe SoandSo
Ok, So I am looking for a little direction and possibly some ideas on how
to handle a problem that has happened on an sbs2k server. First a little
history. The server has a raid 5 drive configuration. Last week one of
the three drives failed. I promptly bought a replacement drive and put it
into the system. About 5 percent into the rebuild the rebuild failed,
the replacement drive went to a failed state, and another of the drives
also failed. So I had two drives in the failed state! I called Dell (it
is a 6 year old dell server), they said that they have seen when older
drives rebuild they sometimes fail because of a timeout issue that is
fixed in newer firmware for the harddrives. So, long story short, we
forced the newly failed drive back on, we used a server2k3 recovery
console to run chkdsk, this made the server again bootable (it would blue
screen during bootup before we ran chkdsk), then we updated the harddrive
firmware on all three harddrives and attempt another rebuild. This
rebuild worked and everything appeared very good. However I am now
noticing errors in the application log about exchange. I will attach the
errors to the end of this message (sorry for the long post). Some of the
errors happen during bootup. I've tried rebooting a few times but the
same errors occur. Other errors happen at night. I've only seen this one
night, tonight will be the next night to see if it happens again.
Backupexec also gave an error.
What do you think I am up against? And what do you suggest I should I do.
My last good backup was Friday Night the 5th.
.Restore the whole system from backup .. con's - I hope it works well,
what do I do with all the changes/new files/new email since Friday.
.Restore just exchange - con's - how do I preserve changes made since
Friday? Will this really fix it.
.Call Microsoft support and try to repair the existing database - will
there be more problems/missing things down the road.
On top of all this, I would like to somehow snapshot the entire system on
to a spare harddrive(s) so that if something goes wrong in the recovery
procedure I can go back to the current state. Does anyone have any
recommendation on how to do this? Mirror? Swap/in/out a few drives into
the raid? Image software to usb drive?
Your suggestions are greatly appreciated. I will check back tomorrow to
see what suggestions you might have.
Thanks Again!
======================================================================
Media Name: "Media created 7/20/2004 09:31:32 AM"
Backup of "\\NETSERVER\Microsoft Exchange Mailboxes "
Backup set #1 on storage media #1
Backup set description: "Nightly Full Backup"
Backup Type: FULL - Back Up Files - Reset Archive Bit
Backup started on 6/8/2009 at 9:00:29 PM.
Backup completed on 6/8/2009 at 11:40:33 PM.
Backed up 116973 mail message(s) in 1602 folder(s) in 37 mailbox(es)
Processed 9,926,872,461 bytes in 2 hours, 40 minutes, and 4 seconds.
Throughput rate: 59.1 MB/min
----------------------------------------------------------------------
Media Name: "Media created 7/20/2004 09:31:32 AM"
Backup of "\\NETSERVER\Microsoft Information Store\First Storage Group "
Backup set #2 on storage media #1
Backup set description: "Nightly Full Backup"
Backup Type: FULL - Database & Logs (flush committed logs)
Backup started on 6/8/2009 at 11:40:35 PM.
Access denied to database .
^ ^ ^ ^ ^
WARNING: "Mailbox Store (NETSERVER)" is a corrupt file.
This file cannot verify.
^ ^ ^ ^ ^
Log files
Backup completed on 6/9/2009 at 12:54:33 AM.
Backed up 2 Exchange Server store(s)
Backed up Exchange Server logs
1 corrupt file was backed up
Processed 16,759,227,384 bytes in 1 hour, 13 minutes, and 58 seconds.
Throughput rate: 216.1 MB/min
----------------------------------------------------------------------
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 302
Date: 6/9/2009
Time: 5:04:14 PM
User: N/A
Computer: NETSERVER
Information Store (4584) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: The
database engine has successfully completed recovery steps.
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: MSExchangeIS Public Store
Event Category: General
Event ID: 1216
Date: 6/9/2009
Time: 5:04:21 PM
User: N/A
Computer: NETSERVER
The information store database "First Storage Group\Public Folder Store
(NETSERVER)" is limited to 16384 MB.
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 301
Date: 6/9/2009
Time: 5:04:08 PM
User: N/A
Computer: NETSERVER
Information Store (4584) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: The
database engine has begun replaying logfile E:\Program
Files\Exchsrvr\mdbdata\E00.log.
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 301
Date: 6/9/2009
Time: 5:04:00 PM
User: N/A
Computer: NETSERVER
Information Store (4584) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: The
database engine has begun replaying logfile E:\Program
Files\Exchsrvr\mdbdata\E000E656.log.
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 300
Date: 6/9/2009
Time: 5:02:41 PM
User: N/A
Computer: NETSERVER
Information Store (4584) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: The
database engine is initiating recovery steps.
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: General
Event ID: 102
Date: 6/9/2009
Time: 5:02:35 PM
User: N/A
Computer: NETSERVER
Information Store (4584) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: The
database engine started a new instance (0).
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Warning
Event Source: Perflib
Event Category: None
Event ID: 1016
Date: 6/9/2009
Time: 5:02:33 PM
User: N/A
Computer: NETSERVER
The data buffer created for the "EXOLEDB" service in the "C:\Program
Files\Exchsrvr\bin\exodbpc.dll" library is not aligned on an 8-byte
boundary. This may cause problems for applications that are trying to
read the performance data buffer. Contact the manufacturer of this
library or service to have this problem corrected or to get a newer
version of this library.
0000: 0c 19 8a 00 04 15 00 00 ..S.....
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 213
Date: 6/9/2009
Time: 12:54:34 AM
User: N/A
Computer: NETSERVER
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: The backup
procedure has been successfully completed.
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 225
Date: 6/9/2009
Time: 12:54:34 AM
User: N/A
Computer: NETSERVER
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: No log
files can be truncated.
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 223
Date: 6/9/2009
Time: 12:54:17 AM
User: N/A
Computer: NETSERVER
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: Starting
the backup of log files (range E:\Program
Files\Exchsrvr\mdbdata\E000E63B.log - E:\Program
Files\Exchsrvr\mdbdata\E000E63E.log).
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 221
Date: 6/9/2009
Time: 12:54:17 AM
User: N/A
Computer: NETSERVER
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: Ending the
backup of the file E:\Program Files\Exchsrvr\mdbdata\pub1.stm.
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 222
Date: 6/9/2009
Time: 12:41:43 AM
User: N/A
Computer: NETSERVER
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: Ending the
backup of the file E:\Program Files\Exchsrvr\mdbdata\priv1.edb. Not all
data in the file has been read (read 11242700800 bytes out of 16215977984
bytes).
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Error
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 217
Date: 6/9/2009
Time: 12:41:43 AM
User: N/A
Computer: NETSERVER
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: Error
(-1018) during backup of a database (file E:\Program
Files\Exchsrvr\mdbdata\priv1.edb). The database will be unable to restore.
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Error
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 474
Date: 6/9/2009
Time: 12:41:43 AM
User: N/A
Computer: NETSERVER
Information Store (4628) The database page read from the file "E:\Program
Files\Exchsrvr\mdbdata\priv1.edb" at offset 11242729472
(0x000000029e1e7000) for 4096 (0x00001000) bytes failed verification due
to a page checksum mismatch. The expected checksum was 4205941953
(0xfab194c1) and the actual checksum was 571156525 (0x220b282d). The read
operation will fail with error -1018 (0xfffffc06). If this condition
persists then please restore the database from a previous backup.
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Error
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 474
Date: 6/9/2009
Time: 12:41:41 AM
User: N/A
Computer: NETSERVER
Information Store (4628) The database page read from the file "E:\Program
Files\Exchsrvr\mdbdata\priv1.edb" at offset 11242729472
(0x000000029e1e7000) for 4096 (0x00001000) bytes failed verification due
to a page checksum mismatch. The expected checksum was 4205941953
(0xfab194c1) and the actual checksum was 571156525 (0x220b282d). The read
operation will fail with error -1018 (0xfffffc06). If this condition
persists then please restore the database from a previous backup.
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 220
Date: 6/8/2009
Time: 11:40:47 PM
User: N/A
Computer: NETSERVER
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: Begining
the backup of the file E:\Program Files\Exchsrvr\mdbdata\priv1.edb (size
15464 Mb).
For more information, click http://www.microsoft.com/contentredirect.asp.
Joe SoandSo
2009-06-10 16:47:22 UTC
Permalink
Are you sure that is my (only) problem?? I sure didn't see these errors
earlier..

Also, last night somebody forgot to put in the backup tape, so we didn't see
the errors that happened after midnight.. I think something in the backup
process triggers them.

Thanks,
Joe.
Post by Merv Porter [SBS-MVP]
It looks like you've hit the 16GB database limit for your Exchange store.
You may be able to temporarily increase the limit to 17GB but you'll need to
How to temporarily increase the Exchange 2000 16-gigabyte database size
limit
http://support.microsoft.com/kb/813051
Exceeding the 16GB Limit in Exchange 2000/2003
http://thelazyadmin.com/blogs/thelazyadmin/archive/2005/06/22/Exceeding-the-16GB-Limit-in-Exchange-2000_2F00_2003.aspx
Exchange Standard and the 16GB Limit
http://hellomate.typepad.com/exchange/2004/03/exchange_standa.html
It may be time to consider upgrading to SBS 2008 (or at least SBS 2003).
--
Merv Porter [SBS-MVP]
============================
Post by Joe SoandSo
Ok, So I am looking for a little direction and possibly some ideas on how
to handle a problem that has happened on an sbs2k server. First a
little history. The server has a raid 5 drive configuration. Last week
one of the three drives failed. I promptly bought a replacement drive
and put it into the system. About 5 percent into the rebuild the
rebuild failed, the replacement drive went to a failed state, and another
of the drives also failed. So I had two drives in the failed state! I
called Dell (it is a 6 year old dell server), they said that they have
seen when older drives rebuild they sometimes fail because of a timeout
issue that is fixed in newer firmware for the harddrives. So, long story
short, we forced the newly failed drive back on, we used a server2k3
recovery console to run chkdsk, this made the server again bootable (it
would blue screen during bootup before we ran chkdsk), then we updated
the harddrive firmware on all three harddrives and attempt another
rebuild. This rebuild worked and everything appeared very good.
However I am now noticing errors in the application log about exchange.
I will attach the errors to the end of this message (sorry for the long
post). Some of the errors happen during bootup. I've tried rebooting a
few times but the same errors occur. Other errors happen at night. I've
only seen this one night, tonight will be the next night to see if it
happens again. Backupexec also gave an error.
What do you think I am up against? And what do you suggest I should I
do. My last good backup was Friday Night the 5th.
.Restore the whole system from backup .. con's - I hope it works well,
what do I do with all the changes/new files/new email since Friday.
.Restore just exchange - con's - how do I preserve changes made since
Friday? Will this really fix it.
.Call Microsoft support and try to repair the existing database - will
there be more problems/missing things down the road.
On top of all this, I would like to somehow snapshot the entire system on
to a spare harddrive(s) so that if something goes wrong in the recovery
procedure I can go back to the current state. Does anyone have any
recommendation on how to do this? Mirror? Swap/in/out a few drives into
the raid? Image software to usb drive?
Your suggestions are greatly appreciated. I will check back tomorrow to
see what suggestions you might have.
Thanks Again!
======================================================================
Media Name: "Media created 7/20/2004 09:31:32 AM"
Backup of "\\NETSERVER\Microsoft Exchange Mailboxes "
Backup set #1 on storage media #1
Backup set description: "Nightly Full Backup"
Backup Type: FULL - Back Up Files - Reset Archive Bit
Backup started on 6/8/2009 at 9:00:29 PM.
Backup completed on 6/8/2009 at 11:40:33 PM.
Backed up 116973 mail message(s) in 1602 folder(s) in 37 mailbox(es)
Processed 9,926,872,461 bytes in 2 hours, 40 minutes, and 4 seconds.
Throughput rate: 59.1 MB/min
----------------------------------------------------------------------
Media Name: "Media created 7/20/2004 09:31:32 AM"
Backup of "\\NETSERVER\Microsoft Information Store\First Storage Group "
Backup set #2 on storage media #1
Backup set description: "Nightly Full Backup"
Backup Type: FULL - Database & Logs (flush committed logs)
Backup started on 6/8/2009 at 11:40:35 PM.
Access denied to database .
^ ^ ^ ^ ^
WARNING: "Mailbox Store (NETSERVER)" is a corrupt file.
This file cannot verify.
^ ^ ^ ^ ^
Log files
Backup completed on 6/9/2009 at 12:54:33 AM.
Backed up 2 Exchange Server store(s)
Backed up Exchange Server logs
1 corrupt file was backed up
Processed 16,759,227,384 bytes in 1 hour, 13 minutes, and 58 seconds.
Throughput rate: 216.1 MB/min
----------------------------------------------------------------------
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 302
Date: 6/9/2009
Time: 5:04:14 PM
User: N/A
Computer: NETSERVER
Information Store (4584) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: The
database engine has successfully completed recovery steps.
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: MSExchangeIS Public Store
Event Category: General
Event ID: 1216
Date: 6/9/2009
Time: 5:04:21 PM
User: N/A
Computer: NETSERVER
The information store database "First Storage Group\Public Folder Store
(NETSERVER)" is limited to 16384 MB.
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 301
Date: 6/9/2009
Time: 5:04:08 PM
User: N/A
Computer: NETSERVER
Information Store (4584) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: The
database engine has begun replaying logfile E:\Program
Files\Exchsrvr\mdbdata\E00.log.
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 301
Date: 6/9/2009
Time: 5:04:00 PM
User: N/A
Computer: NETSERVER
Information Store (4584) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: The
database engine has begun replaying logfile E:\Program
Files\Exchsrvr\mdbdata\E000E656.log.
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 300
Date: 6/9/2009
Time: 5:02:41 PM
User: N/A
Computer: NETSERVER
Information Store (4584) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: The
database engine is initiating recovery steps.
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: General
Event ID: 102
Date: 6/9/2009
Time: 5:02:35 PM
User: N/A
Computer: NETSERVER
Information Store (4584) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: The
database engine started a new instance (0).
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Warning
Event Source: Perflib
Event Category: None
Event ID: 1016
Date: 6/9/2009
Time: 5:02:33 PM
User: N/A
Computer: NETSERVER
The data buffer created for the "EXOLEDB" service in the "C:\Program
Files\Exchsrvr\bin\exodbpc.dll" library is not aligned on an 8-byte
boundary. This may cause problems for applications that are trying to
read the performance data buffer. Contact the manufacturer of this
library or service to have this problem corrected or to get a newer
version of this library.
0000: 0c 19 8a 00 04 15 00 00 ..S.....
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 213
Date: 6/9/2009
Time: 12:54:34 AM
User: N/A
Computer: NETSERVER
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: The backup
procedure has been successfully completed.
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 225
Date: 6/9/2009
Time: 12:54:34 AM
User: N/A
Computer: NETSERVER
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: No log
files can be truncated.
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 223
Date: 6/9/2009
Time: 12:54:17 AM
User: N/A
Computer: NETSERVER
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: Starting
the backup of log files (range E:\Program
Files\Exchsrvr\mdbdata\E000E63B.log - E:\Program
Files\Exchsrvr\mdbdata\E000E63E.log).
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 221
Date: 6/9/2009
Time: 12:54:17 AM
User: N/A
Computer: NETSERVER
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: Ending the
backup of the file E:\Program Files\Exchsrvr\mdbdata\pub1.stm.
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 222
Date: 6/9/2009
Time: 12:41:43 AM
User: N/A
Computer: NETSERVER
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: Ending the
backup of the file E:\Program Files\Exchsrvr\mdbdata\priv1.edb. Not all
data in the file has been read (read 11242700800 bytes out of 16215977984
bytes).
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Error
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 217
Date: 6/9/2009
Time: 12:41:43 AM
User: N/A
Computer: NETSERVER
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: Error
(-1018) during backup of a database (file E:\Program
Files\Exchsrvr\mdbdata\priv1.edb). The database will be unable to restore.
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Error
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 474
Date: 6/9/2009
Time: 12:41:43 AM
User: N/A
Computer: NETSERVER
Information Store (4628) The database page read from the file "E:\Program
Files\Exchsrvr\mdbdata\priv1.edb" at offset 11242729472
(0x000000029e1e7000) for 4096 (0x00001000) bytes failed verification due
to a page checksum mismatch. The expected checksum was 4205941953
(0xfab194c1) and the actual checksum was 571156525 (0x220b282d). The
read operation will fail with error -1018 (0xfffffc06). If this
condition persists then please restore the database from a previous
backup.
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Error
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 474
Date: 6/9/2009
Time: 12:41:41 AM
User: N/A
Computer: NETSERVER
Information Store (4628) The database page read from the file "E:\Program
Files\Exchsrvr\mdbdata\priv1.edb" at offset 11242729472
(0x000000029e1e7000) for 4096 (0x00001000) bytes failed verification due
to a page checksum mismatch. The expected checksum was 4205941953
(0xfab194c1) and the actual checksum was 571156525 (0x220b282d). The
read operation will fail with error -1018 (0xfffffc06). If this
condition persists then please restore the database from a previous
backup.
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 220
Date: 6/8/2009
Time: 11:40:47 PM
User: N/A
Computer: NETSERVER
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: Begining
the backup of the file E:\Program Files\Exchsrvr\mdbdata\priv1.edb (size
15464 Mb).
For more information, click http://www.microsoft.com/contentredirect.asp.
Merv Porter [SBS-MVP]
2009-06-10 17:15:02 UTC
Permalink
It may not be your *only* problem, but it is a problem that needs to be
corrected if you've trully hit the 16GB limit.
--
Merv Porter [SBS-MVP]
============================
Post by Joe SoandSo
Are you sure that is my (only) problem?? I sure didn't see these errors
earlier..
Also, last night somebody forgot to put in the backup tape, so we didn't
see the errors that happened after midnight.. I think something in the
backup process triggers them.
Thanks,
Joe.
Post by Merv Porter [SBS-MVP]
It looks like you've hit the 16GB database limit for your Exchange store.
You may be able to temporarily increase the limit to 17GB but you'll need to
How to temporarily increase the Exchange 2000 16-gigabyte database size
limit
http://support.microsoft.com/kb/813051
Exceeding the 16GB Limit in Exchange 2000/2003
http://thelazyadmin.com/blogs/thelazyadmin/archive/2005/06/22/Exceeding-the-16GB-Limit-in-Exchange-2000_2F00_2003.aspx
Exchange Standard and the 16GB Limit
http://hellomate.typepad.com/exchange/2004/03/exchange_standa.html
It may be time to consider upgrading to SBS 2008 (or at least SBS 2003).
--
Merv Porter [SBS-MVP]
============================
Post by Joe SoandSo
Ok, So I am looking for a little direction and possibly some ideas on
how to handle a problem that has happened on an sbs2k server. First a
little history. The server has a raid 5 drive configuration. Last week
one of the three drives failed. I promptly bought a replacement drive
and put it into the system. About 5 percent into the rebuild the
rebuild failed, the replacement drive went to a failed state, and
another of the drives also failed. So I had two drives in the failed
state! I called Dell (it is a 6 year old dell server), they said that
they have seen when older drives rebuild they sometimes fail because of
a timeout issue that is fixed in newer firmware for the harddrives. So,
long story short, we forced the newly failed drive back on, we used a
server2k3 recovery console to run chkdsk, this made the server again
bootable (it would blue screen during bootup before we ran chkdsk), then
we updated the harddrive firmware on all three harddrives and attempt
another rebuild. This rebuild worked and everything appeared very good.
However I am now noticing errors in the application log about exchange.
I will attach the errors to the end of this message (sorry for the long
post). Some of the errors happen during bootup. I've tried rebooting
a few times but the same errors occur. Other errors happen at night.
I've only seen this one night, tonight will be the next night to see if
it happens again. Backupexec also gave an error.
What do you think I am up against? And what do you suggest I should I
do. My last good backup was Friday Night the 5th.
.Restore the whole system from backup .. con's - I hope it works well,
what do I do with all the changes/new files/new email since Friday.
.Restore just exchange - con's - how do I preserve changes made since
Friday? Will this really fix it.
.Call Microsoft support and try to repair the existing database - will
there be more problems/missing things down the road.
On top of all this, I would like to somehow snapshot the entire system
on to a spare harddrive(s) so that if something goes wrong in the
recovery procedure I can go back to the current state. Does anyone have
any recommendation on how to do this? Mirror? Swap/in/out a few drives
into the raid? Image software to usb drive?
Your suggestions are greatly appreciated. I will check back tomorrow to
see what suggestions you might have.
Thanks Again!
======================================================================
Media Name: "Media created 7/20/2004 09:31:32 AM"
Backup of "\\NETSERVER\Microsoft Exchange Mailboxes "
Backup set #1 on storage media #1
Backup set description: "Nightly Full Backup"
Backup Type: FULL - Back Up Files - Reset Archive Bit
Backup started on 6/8/2009 at 9:00:29 PM.
Backup completed on 6/8/2009 at 11:40:33 PM.
Backed up 116973 mail message(s) in 1602 folder(s) in 37 mailbox(es)
Processed 9,926,872,461 bytes in 2 hours, 40 minutes, and 4 seconds.
Throughput rate: 59.1 MB/min
----------------------------------------------------------------------
Media Name: "Media created 7/20/2004 09:31:32 AM"
Backup of "\\NETSERVER\Microsoft Information Store\First Storage Group "
Backup set #2 on storage media #1
Backup set description: "Nightly Full Backup"
Backup Type: FULL - Database & Logs (flush committed logs)
Backup started on 6/8/2009 at 11:40:35 PM.
Access denied to database .
^ ^ ^ ^ ^
WARNING: "Mailbox Store (NETSERVER)" is a corrupt file.
This file cannot verify.
^ ^ ^ ^ ^
Log files
Backup completed on 6/9/2009 at 12:54:33 AM.
Backed up 2 Exchange Server store(s)
Backed up Exchange Server logs
1 corrupt file was backed up
Processed 16,759,227,384 bytes in 1 hour, 13 minutes, and 58 seconds.
Throughput rate: 216.1 MB/min
----------------------------------------------------------------------
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 302
Date: 6/9/2009
Time: 5:04:14 PM
User: N/A
Computer: NETSERVER
Information Store (4584) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: The
database engine has successfully completed recovery steps.
For more information, click
http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: MSExchangeIS Public Store
Event Category: General
Event ID: 1216
Date: 6/9/2009
Time: 5:04:21 PM
User: N/A
Computer: NETSERVER
The information store database "First Storage Group\Public Folder Store
(NETSERVER)" is limited to 16384 MB.
For more information, click
http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 301
Date: 6/9/2009
Time: 5:04:08 PM
User: N/A
Computer: NETSERVER
Information Store (4584) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: The
database engine has begun replaying logfile E:\Program
Files\Exchsrvr\mdbdata\E00.log.
For more information, click
http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 301
Date: 6/9/2009
Time: 5:04:00 PM
User: N/A
Computer: NETSERVER
Information Store (4584) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: The
database engine has begun replaying logfile E:\Program
Files\Exchsrvr\mdbdata\E000E656.log.
For more information, click
http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 300
Date: 6/9/2009
Time: 5:02:41 PM
User: N/A
Computer: NETSERVER
Information Store (4584) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: The
database engine is initiating recovery steps.
For more information, click
http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: General
Event ID: 102
Date: 6/9/2009
Time: 5:02:35 PM
User: N/A
Computer: NETSERVER
Information Store (4584) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: The
database engine started a new instance (0).
For more information, click
http://www.microsoft.com/contentredirect.asp.
Event Type: Warning
Event Source: Perflib
Event Category: None
Event ID: 1016
Date: 6/9/2009
Time: 5:02:33 PM
User: N/A
Computer: NETSERVER
The data buffer created for the "EXOLEDB" service in the "C:\Program
Files\Exchsrvr\bin\exodbpc.dll" library is not aligned on an 8-byte
boundary. This may cause problems for applications that are trying to
read the performance data buffer. Contact the manufacturer of this
library or service to have this problem corrected or to get a newer
version of this library.
0000: 0c 19 8a 00 04 15 00 00 ..S.....
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 213
Date: 6/9/2009
Time: 12:54:34 AM
User: N/A
Computer: NETSERVER
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: The
backup procedure has been successfully completed.
For more information, click
http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 225
Date: 6/9/2009
Time: 12:54:34 AM
User: N/A
Computer: NETSERVER
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: No log
files can be truncated.
For more information, click
http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 223
Date: 6/9/2009
Time: 12:54:17 AM
User: N/A
Computer: NETSERVER
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: Starting
the backup of log files (range E:\Program
Files\Exchsrvr\mdbdata\E000E63B.log - E:\Program
Files\Exchsrvr\mdbdata\E000E63E.log).
For more information, click
http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 221
Date: 6/9/2009
Time: 12:54:17 AM
User: N/A
Computer: NETSERVER
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: Ending
the backup of the file E:\Program Files\Exchsrvr\mdbdata\pub1.stm.
For more information, click
http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 222
Date: 6/9/2009
Time: 12:41:43 AM
User: N/A
Computer: NETSERVER
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: Ending
the backup of the file E:\Program Files\Exchsrvr\mdbdata\priv1.edb. Not
all data in the file has been read (read 11242700800 bytes out of
16215977984 bytes).
For more information, click
http://www.microsoft.com/contentredirect.asp.
Event Type: Error
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 217
Date: 6/9/2009
Time: 12:41:43 AM
User: N/A
Computer: NETSERVER
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: Error
(-1018) during backup of a database (file E:\Program
Files\Exchsrvr\mdbdata\priv1.edb). The database will be unable to restore.
For more information, click
http://www.microsoft.com/contentredirect.asp.
Event Type: Error
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 474
Date: 6/9/2009
Time: 12:41:43 AM
User: N/A
Computer: NETSERVER
Information Store (4628) The database page read from the file
"E:\Program Files\Exchsrvr\mdbdata\priv1.edb" at offset 11242729472
(0x000000029e1e7000) for 4096 (0x00001000) bytes failed verification due
to a page checksum mismatch. The expected checksum was 4205941953
(0xfab194c1) and the actual checksum was 571156525 (0x220b282d). The
read operation will fail with error -1018 (0xfffffc06). If this
condition persists then please restore the database from a previous
backup.
For more information, click
http://www.microsoft.com/contentredirect.asp.
Event Type: Error
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 474
Date: 6/9/2009
Time: 12:41:41 AM
User: N/A
Computer: NETSERVER
Information Store (4628) The database page read from the file
"E:\Program Files\Exchsrvr\mdbdata\priv1.edb" at offset 11242729472
(0x000000029e1e7000) for 4096 (0x00001000) bytes failed verification due
to a page checksum mismatch. The expected checksum was 4205941953
(0xfab194c1) and the actual checksum was 571156525 (0x220b282d). The
read operation will fail with error -1018 (0xfffffc06). If this
condition persists then please restore the database from a previous
backup.
For more information, click
http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 220
Date: 6/8/2009
Time: 11:40:47 PM
User: N/A
Computer: NETSERVER
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: Begining
the backup of the file E:\Program Files\Exchsrvr\mdbdata\priv1.edb (size
15464 Mb).
For more information, click
http://www.microsoft.com/contentredirect.asp.
Joe SoandSo
2009-06-10 23:07:05 UTC
Permalink
I'll check first thing in the AM. If I remember right, the error for when
we hit the limit is different and the store dismounts.

I still need advice on the other items, if anyone has any advice, I would
sure appreciate it.

-Thanks.
Post by Merv Porter [SBS-MVP]
It may not be your *only* problem, but it is a problem that needs to be
corrected if you've trully hit the 16GB limit.
--
Merv Porter [SBS-MVP]
============================
Post by Joe SoandSo
Are you sure that is my (only) problem?? I sure didn't see these errors
earlier..
Also, last night somebody forgot to put in the backup tape, so we didn't
see the errors that happened after midnight.. I think something in the
backup process triggers them.
Thanks,
Joe.
Post by Merv Porter [SBS-MVP]
It looks like you've hit the 16GB database limit for your Exchange
store. You may be able to temporarily increase the limit to 17GB but
you'll need to
How to temporarily increase the Exchange 2000 16-gigabyte database size
limit
http://support.microsoft.com/kb/813051
Exceeding the 16GB Limit in Exchange 2000/2003
http://thelazyadmin.com/blogs/thelazyadmin/archive/2005/06/22/Exceeding-the-16GB-Limit-in-Exchange-2000_2F00_2003.aspx
Exchange Standard and the 16GB Limit
http://hellomate.typepad.com/exchange/2004/03/exchange_standa.html
It may be time to consider upgrading to SBS 2008 (or at least SBS 2003).
--
Merv Porter [SBS-MVP]
============================
Post by Joe SoandSo
Ok, So I am looking for a little direction and possibly some ideas on
how to handle a problem that has happened on an sbs2k server. First a
little history. The server has a raid 5 drive configuration. Last
week one of the three drives failed. I promptly bought a replacement
drive and put it into the system. About 5 percent into the rebuild
the rebuild failed, the replacement drive went to a failed state, and
another of the drives also failed. So I had two drives in the failed
state! I called Dell (it is a 6 year old dell server), they said that
they have seen when older drives rebuild they sometimes fail because of
a timeout issue that is fixed in newer firmware for the harddrives.
So, long story short, we forced the newly failed drive back on, we used
a server2k3 recovery console to run chkdsk, this made the server again
bootable (it would blue screen during bootup before we ran chkdsk),
then we updated the harddrive firmware on all three harddrives and
attempt another rebuild. This rebuild worked and everything appeared
very good. However I am now noticing errors in the application log
about exchange. I will attach the errors to the end of this message
(sorry for the long post). Some of the errors happen during bootup.
I've tried rebooting a few times but the same errors occur. Other
errors happen at night. I've only seen this one night, tonight will be
the next night to see if it happens again. Backupexec also gave an
error.
What do you think I am up against? And what do you suggest I should I
do. My last good backup was Friday Night the 5th.
.Restore the whole system from backup .. con's - I hope it works well,
what do I do with all the changes/new files/new email since Friday.
.Restore just exchange - con's - how do I preserve changes made since
Friday? Will this really fix it.
.Call Microsoft support and try to repair the existing database - will
there be more problems/missing things down the road.
On top of all this, I would like to somehow snapshot the entire system
on to a spare harddrive(s) so that if something goes wrong in the
recovery procedure I can go back to the current state. Does anyone
have any recommendation on how to do this? Mirror? Swap/in/out a few
drives into the raid? Image software to usb drive?
Your suggestions are greatly appreciated. I will check back tomorrow
to see what suggestions you might have.
Thanks Again!
======================================================================
Media Name: "Media created 7/20/2004 09:31:32 AM"
Backup of "\\NETSERVER\Microsoft Exchange Mailboxes "
Backup set #1 on storage media #1
Backup set description: "Nightly Full Backup"
Backup Type: FULL - Back Up Files - Reset Archive Bit
Backup started on 6/8/2009 at 9:00:29 PM.
Backup completed on 6/8/2009 at 11:40:33 PM.
Backed up 116973 mail message(s) in 1602 folder(s) in 37 mailbox(es)
Processed 9,926,872,461 bytes in 2 hours, 40 minutes, and 4 seconds.
Throughput rate: 59.1 MB/min
----------------------------------------------------------------------
Media Name: "Media created 7/20/2004 09:31:32 AM"
Backup of "\\NETSERVER\Microsoft Information Store\First Storage Group "
Backup set #2 on storage media #1
Backup set description: "Nightly Full Backup"
Backup Type: FULL - Database & Logs (flush committed logs)
Backup started on 6/8/2009 at 11:40:35 PM.
Access denied to database .
^ ^ ^ ^ ^
WARNING: "Mailbox Store (NETSERVER)" is a corrupt file.
This file cannot verify.
^ ^ ^ ^ ^
Log files
Backup completed on 6/9/2009 at 12:54:33 AM.
Backed up 2 Exchange Server store(s)
Backed up Exchange Server logs
1 corrupt file was backed up
Processed 16,759,227,384 bytes in 1 hour, 13 minutes, and 58 seconds.
Throughput rate: 216.1 MB/min
----------------------------------------------------------------------
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 302
Date: 6/9/2009
Time: 5:04:14 PM
User: N/A
Computer: NETSERVER
Information Store (4584) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: The
database engine has successfully completed recovery steps.
For more information, click
http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: MSExchangeIS Public Store
Event Category: General
Event ID: 1216
Date: 6/9/2009
Time: 5:04:21 PM
User: N/A
Computer: NETSERVER
The information store database "First Storage Group\Public Folder Store
(NETSERVER)" is limited to 16384 MB.
For more information, click
http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 301
Date: 6/9/2009
Time: 5:04:08 PM
User: N/A
Computer: NETSERVER
Information Store (4584) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: The
database engine has begun replaying logfile E:\Program
Files\Exchsrvr\mdbdata\E00.log.
For more information, click
http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 301
Date: 6/9/2009
Time: 5:04:00 PM
User: N/A
Computer: NETSERVER
Information Store (4584) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: The
database engine has begun replaying logfile E:\Program
Files\Exchsrvr\mdbdata\E000E656.log.
For more information, click
http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 300
Date: 6/9/2009
Time: 5:02:41 PM
User: N/A
Computer: NETSERVER
Information Store (4584) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: The
database engine is initiating recovery steps.
For more information, click
http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: General
Event ID: 102
Date: 6/9/2009
Time: 5:02:35 PM
User: N/A
Computer: NETSERVER
Information Store (4584) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: The
database engine started a new instance (0).
For more information, click
http://www.microsoft.com/contentredirect.asp.
Event Type: Warning
Event Source: Perflib
Event Category: None
Event ID: 1016
Date: 6/9/2009
Time: 5:02:33 PM
User: N/A
Computer: NETSERVER
The data buffer created for the "EXOLEDB" service in the "C:\Program
Files\Exchsrvr\bin\exodbpc.dll" library is not aligned on an 8-byte
boundary. This may cause problems for applications that are trying to
read the performance data buffer. Contact the manufacturer of this
library or service to have this problem corrected or to get a newer
version of this library.
0000: 0c 19 8a 00 04 15 00 00 ..S.....
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 213
Date: 6/9/2009
Time: 12:54:34 AM
User: N/A
Computer: NETSERVER
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: The
backup procedure has been successfully completed.
For more information, click
http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 225
Date: 6/9/2009
Time: 12:54:34 AM
User: N/A
Computer: NETSERVER
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: No log
files can be truncated.
For more information, click
http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 223
Date: 6/9/2009
Time: 12:54:17 AM
User: N/A
Computer: NETSERVER
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: Starting
the backup of log files (range E:\Program
Files\Exchsrvr\mdbdata\E000E63B.log - E:\Program
Files\Exchsrvr\mdbdata\E000E63E.log).
For more information, click
http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 221
Date: 6/9/2009
Time: 12:54:17 AM
User: N/A
Computer: NETSERVER
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: Ending
the backup of the file E:\Program Files\Exchsrvr\mdbdata\pub1.stm.
For more information, click
http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 222
Date: 6/9/2009
Time: 12:41:43 AM
User: N/A
Computer: NETSERVER
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: Ending
the backup of the file E:\Program Files\Exchsrvr\mdbdata\priv1.edb. Not
all data in the file has been read (read 11242700800 bytes out of
16215977984 bytes).
For more information, click
http://www.microsoft.com/contentredirect.asp.
Event Type: Error
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 217
Date: 6/9/2009
Time: 12:41:43 AM
User: N/A
Computer: NETSERVER
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: Error
(-1018) during backup of a database (file E:\Program
Files\Exchsrvr\mdbdata\priv1.edb). The database will be unable to restore.
For more information, click
http://www.microsoft.com/contentredirect.asp.
Event Type: Error
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 474
Date: 6/9/2009
Time: 12:41:43 AM
User: N/A
Computer: NETSERVER
Information Store (4628) The database page read from the file
"E:\Program Files\Exchsrvr\mdbdata\priv1.edb" at offset 11242729472
(0x000000029e1e7000) for 4096 (0x00001000) bytes failed verification
due to a page checksum mismatch. The expected checksum was 4205941953
(0xfab194c1) and the actual checksum was 571156525 (0x220b282d). The
read operation will fail with error -1018 (0xfffffc06). If this
condition persists then please restore the database from a previous
backup.
For more information, click
http://www.microsoft.com/contentredirect.asp.
Event Type: Error
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 474
Date: 6/9/2009
Time: 12:41:41 AM
User: N/A
Computer: NETSERVER
Information Store (4628) The database page read from the file
"E:\Program Files\Exchsrvr\mdbdata\priv1.edb" at offset 11242729472
(0x000000029e1e7000) for 4096 (0x00001000) bytes failed verification
due to a page checksum mismatch. The expected checksum was 4205941953
(0xfab194c1) and the actual checksum was 571156525 (0x220b282d). The
read operation will fail with error -1018 (0xfffffc06). If this
condition persists then please restore the database from a previous
backup.
For more information, click
http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 220
Date: 6/8/2009
Time: 11:40:47 PM
User: N/A
Computer: NETSERVER
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: Begining
the backup of the file E:\Program Files\Exchsrvr\mdbdata\priv1.edb
(size 15464 Mb).
For more information, click
http://www.microsoft.com/contentredirect.asp.
Loading...