Joe SoandSo
2009-06-10 05:26:27 UTC
Ok, So I am looking for a little direction and possibly some ideas on how to
handle a problem that has happened on an sbs2k server. First a little
history. The server has a raid 5 drive configuration. Last week one of the
three drives failed. I promptly bought a replacement drive and put it into
the system. About 5 percent into the rebuild the rebuild failed, the
replacement drive went to a failed state, and another of the drives also
failed. So I had two drives in the failed state! I called Dell (it is a 6
year old dell server), they said that they have seen when older drives
rebuild they sometimes fail because of a timeout issue that is fixed in
newer firmware for the harddrives. So, long story short, we forced the
newly failed drive back on, we used a server2k3 recovery console to run
chkdsk, this made the server again bootable (it would blue screen during
bootup before we ran chkdsk), then we updated the harddrive firmware on all
three harddrives and attempt another rebuild. This rebuild worked and
everything appeared very good. However I am now noticing errors in the
application log about exchange. I will attach the errors to the end of this
message (sorry for the long post). Some of the errors happen during
bootup. I've tried rebooting a few times but the same errors occur. Other
errors happen at night. I've only seen this one night, tonight will be the
next night to see if it happens again. Backupexec also gave an error.
What do you think I am up against? And what do you suggest I should I do.
My last good backup was Friday Night the 5th.
Now as I see it I have several options:
.Restore the whole system from backup .. con's - I hope it works well, what
do I do with all the changes/new files/new email since Friday.
.Restore just exchange - con's - how do I preserve changes made since
Friday? Will this really fix it.
.Call Microsoft support and try to repair the existing database - will there
be more problems/missing things down the road.
On top of all this, I would like to somehow snapshot the entire system on to
a spare harddrive(s) so that if something goes wrong in the recovery
procedure I can go back to the current state. Does anyone have any
recommendation on how to do this? Mirror? Swap/in/out a few drives into the
raid? Image software to usb drive?
Your suggestions are greatly appreciated. I will check back tomorrow to see
what suggestions you might have.
Thanks Again!
error messages:
======================================================================
Media Name: "Media created 7/20/2004 09:31:32 AM"
Backup of "\\NETSERVER\Microsoft Exchange Mailboxes "
Backup set #1 on storage media #1
Backup set description: "Nightly Full Backup"
Backup Type: FULL - Back Up Files - Reset Archive Bit
Backup started on 6/8/2009 at 9:00:29 PM.
Backup completed on 6/8/2009 at 11:40:33 PM.
Backed up 116973 mail message(s) in 1602 folder(s) in 37 mailbox(es)
Processed 9,926,872,461 bytes in 2 hours, 40 minutes, and 4 seconds.
Throughput rate: 59.1 MB/min
----------------------------------------------------------------------
Media Name: "Media created 7/20/2004 09:31:32 AM"
Backup of "\\NETSERVER\Microsoft Information Store\First Storage Group "
Backup set #2 on storage media #1
Backup set description: "Nightly Full Backup"
Backup Type: FULL - Database & Logs (flush committed logs)
Backup started on 6/8/2009 at 11:40:35 PM.
Access denied to database .
^ ^ ^ ^ ^
WARNING: "Mailbox Store (NETSERVER)" is a corrupt file.
This file cannot verify.
^ ^ ^ ^ ^
Log files
Backup completed on 6/9/2009 at 12:54:33 AM.
Backed up 2 Exchange Server store(s)
Backed up Exchange Server logs
1 corrupt file was backed up
Processed 16,759,227,384 bytes in 1 hour, 13 minutes, and 58 seconds.
Throughput rate: 216.1 MB/min
----------------------------------------------------------------------
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 302
Date: 6/9/2009
Time: 5:04:14 PM
User: N/A
Computer: NETSERVER
Description:
Information Store (4584) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: The database
engine has successfully completed recovery steps.
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: MSExchangeIS Public Store
Event Category: General
Event ID: 1216
Date: 6/9/2009
Time: 5:04:21 PM
User: N/A
Computer: NETSERVER
Description:
The information store database "First Storage Group\Public Folder Store
(NETSERVER)" is limited to 16384 MB.
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 301
Date: 6/9/2009
Time: 5:04:08 PM
User: N/A
Computer: NETSERVER
Description:
Information Store (4584) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: The database
engine has begun replaying logfile E:\Program
Files\Exchsrvr\mdbdata\E00.log.
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 301
Date: 6/9/2009
Time: 5:04:00 PM
User: N/A
Computer: NETSERVER
Description:
Information Store (4584) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: The database
engine has begun replaying logfile E:\Program
Files\Exchsrvr\mdbdata\E000E656.log.
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 300
Date: 6/9/2009
Time: 5:02:41 PM
User: N/A
Computer: NETSERVER
Description:
Information Store (4584) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: The database
engine is initiating recovery steps.
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: General
Event ID: 102
Date: 6/9/2009
Time: 5:02:35 PM
User: N/A
Computer: NETSERVER
Description:
Information Store (4584) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: The database
engine started a new instance (0).
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Warning
Event Source: Perflib
Event Category: None
Event ID: 1016
Date: 6/9/2009
Time: 5:02:33 PM
User: N/A
Computer: NETSERVER
Description:
The data buffer created for the "EXOLEDB" service in the "C:\Program
Files\Exchsrvr\bin\exodbpc.dll" library is not aligned on an 8-byte
boundary. This may cause problems for applications that are trying to read
the performance data buffer. Contact the manufacturer of this library or
service to have this problem corrected or to get a newer version of this
library.
Data:
0000: 0c 19 8a 00 04 15 00 00 .......
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 213
Date: 6/9/2009
Time: 12:54:34 AM
User: N/A
Computer: NETSERVER
Description:
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: The backup
procedure has been successfully completed.
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 225
Date: 6/9/2009
Time: 12:54:34 AM
User: N/A
Computer: NETSERVER
Description:
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: No log files
can be truncated.
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 223
Date: 6/9/2009
Time: 12:54:17 AM
User: N/A
Computer: NETSERVER
Description:
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: Starting the
backup of log files (range E:\Program Files\Exchsrvr\mdbdata\E000E63B.log -
E:\Program Files\Exchsrvr\mdbdata\E000E63E.log).
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 221
Date: 6/9/2009
Time: 12:54:17 AM
User: N/A
Computer: NETSERVER
Description:
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: Ending the
backup of the file E:\Program Files\Exchsrvr\mdbdata\pub1.stm.
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 222
Date: 6/9/2009
Time: 12:41:43 AM
User: N/A
Computer: NETSERVER
Description:
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: Ending the
backup of the file E:\Program Files\Exchsrvr\mdbdata\priv1.edb. Not all data
in the file has been read (read 11242700800 bytes out of 16215977984 bytes).
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Error
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 217
Date: 6/9/2009
Time: 12:41:43 AM
User: N/A
Computer: NETSERVER
Description:
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: Error (-1018)
during backup of a database (file E:\Program
Files\Exchsrvr\mdbdata\priv1.edb). The database will be unable to restore.
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Error
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 474
Date: 6/9/2009
Time: 12:41:43 AM
User: N/A
Computer: NETSERVER
Description:
Information Store (4628) The database page read from the file "E:\Program
Files\Exchsrvr\mdbdata\priv1.edb" at offset 11242729472 (0x000000029e1e7000)
for 4096 (0x00001000) bytes failed verification due to a page checksum
mismatch. The expected checksum was 4205941953 (0xfab194c1) and the actual
checksum was 571156525 (0x220b282d). The read operation will fail with
error -1018 (0xfffffc06). If this condition persists then please restore
the database from a previous backup.
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Error
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 474
Date: 6/9/2009
Time: 12:41:41 AM
User: N/A
Computer: NETSERVER
Description:
Information Store (4628) The database page read from the file "E:\Program
Files\Exchsrvr\mdbdata\priv1.edb" at offset 11242729472 (0x000000029e1e7000)
for 4096 (0x00001000) bytes failed verification due to a page checksum
mismatch. The expected checksum was 4205941953 (0xfab194c1) and the actual
checksum was 571156525 (0x220b282d). The read operation will fail with
error -1018 (0xfffffc06). If this condition persists then please restore
the database from a previous backup.
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 220
Date: 6/8/2009
Time: 11:40:47 PM
User: N/A
Computer: NETSERVER
Description:
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: Begining the
backup of the file E:\Program Files\Exchsrvr\mdbdata\priv1.edb (size 15464
Mb).
For more information, click http://www.microsoft.com/contentredirect.asp.
handle a problem that has happened on an sbs2k server. First a little
history. The server has a raid 5 drive configuration. Last week one of the
three drives failed. I promptly bought a replacement drive and put it into
the system. About 5 percent into the rebuild the rebuild failed, the
replacement drive went to a failed state, and another of the drives also
failed. So I had two drives in the failed state! I called Dell (it is a 6
year old dell server), they said that they have seen when older drives
rebuild they sometimes fail because of a timeout issue that is fixed in
newer firmware for the harddrives. So, long story short, we forced the
newly failed drive back on, we used a server2k3 recovery console to run
chkdsk, this made the server again bootable (it would blue screen during
bootup before we ran chkdsk), then we updated the harddrive firmware on all
three harddrives and attempt another rebuild. This rebuild worked and
everything appeared very good. However I am now noticing errors in the
application log about exchange. I will attach the errors to the end of this
message (sorry for the long post). Some of the errors happen during
bootup. I've tried rebooting a few times but the same errors occur. Other
errors happen at night. I've only seen this one night, tonight will be the
next night to see if it happens again. Backupexec also gave an error.
What do you think I am up against? And what do you suggest I should I do.
My last good backup was Friday Night the 5th.
Now as I see it I have several options:
.Restore the whole system from backup .. con's - I hope it works well, what
do I do with all the changes/new files/new email since Friday.
.Restore just exchange - con's - how do I preserve changes made since
Friday? Will this really fix it.
.Call Microsoft support and try to repair the existing database - will there
be more problems/missing things down the road.
On top of all this, I would like to somehow snapshot the entire system on to
a spare harddrive(s) so that if something goes wrong in the recovery
procedure I can go back to the current state. Does anyone have any
recommendation on how to do this? Mirror? Swap/in/out a few drives into the
raid? Image software to usb drive?
Your suggestions are greatly appreciated. I will check back tomorrow to see
what suggestions you might have.
Thanks Again!
error messages:
======================================================================
Media Name: "Media created 7/20/2004 09:31:32 AM"
Backup of "\\NETSERVER\Microsoft Exchange Mailboxes "
Backup set #1 on storage media #1
Backup set description: "Nightly Full Backup"
Backup Type: FULL - Back Up Files - Reset Archive Bit
Backup started on 6/8/2009 at 9:00:29 PM.
Backup completed on 6/8/2009 at 11:40:33 PM.
Backed up 116973 mail message(s) in 1602 folder(s) in 37 mailbox(es)
Processed 9,926,872,461 bytes in 2 hours, 40 minutes, and 4 seconds.
Throughput rate: 59.1 MB/min
----------------------------------------------------------------------
Media Name: "Media created 7/20/2004 09:31:32 AM"
Backup of "\\NETSERVER\Microsoft Information Store\First Storage Group "
Backup set #2 on storage media #1
Backup set description: "Nightly Full Backup"
Backup Type: FULL - Database & Logs (flush committed logs)
Backup started on 6/8/2009 at 11:40:35 PM.
Access denied to database .
^ ^ ^ ^ ^
WARNING: "Mailbox Store (NETSERVER)" is a corrupt file.
This file cannot verify.
^ ^ ^ ^ ^
Log files
Backup completed on 6/9/2009 at 12:54:33 AM.
Backed up 2 Exchange Server store(s)
Backed up Exchange Server logs
1 corrupt file was backed up
Processed 16,759,227,384 bytes in 1 hour, 13 minutes, and 58 seconds.
Throughput rate: 216.1 MB/min
----------------------------------------------------------------------
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 302
Date: 6/9/2009
Time: 5:04:14 PM
User: N/A
Computer: NETSERVER
Description:
Information Store (4584) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: The database
engine has successfully completed recovery steps.
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: MSExchangeIS Public Store
Event Category: General
Event ID: 1216
Date: 6/9/2009
Time: 5:04:21 PM
User: N/A
Computer: NETSERVER
Description:
The information store database "First Storage Group\Public Folder Store
(NETSERVER)" is limited to 16384 MB.
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 301
Date: 6/9/2009
Time: 5:04:08 PM
User: N/A
Computer: NETSERVER
Description:
Information Store (4584) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: The database
engine has begun replaying logfile E:\Program
Files\Exchsrvr\mdbdata\E00.log.
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 301
Date: 6/9/2009
Time: 5:04:00 PM
User: N/A
Computer: NETSERVER
Description:
Information Store (4584) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: The database
engine has begun replaying logfile E:\Program
Files\Exchsrvr\mdbdata\E000E656.log.
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 300
Date: 6/9/2009
Time: 5:02:41 PM
User: N/A
Computer: NETSERVER
Description:
Information Store (4584) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: The database
engine is initiating recovery steps.
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: General
Event ID: 102
Date: 6/9/2009
Time: 5:02:35 PM
User: N/A
Computer: NETSERVER
Description:
Information Store (4584) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: The database
engine started a new instance (0).
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Warning
Event Source: Perflib
Event Category: None
Event ID: 1016
Date: 6/9/2009
Time: 5:02:33 PM
User: N/A
Computer: NETSERVER
Description:
The data buffer created for the "EXOLEDB" service in the "C:\Program
Files\Exchsrvr\bin\exodbpc.dll" library is not aligned on an 8-byte
boundary. This may cause problems for applications that are trying to read
the performance data buffer. Contact the manufacturer of this library or
service to have this problem corrected or to get a newer version of this
library.
Data:
0000: 0c 19 8a 00 04 15 00 00 .......
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 213
Date: 6/9/2009
Time: 12:54:34 AM
User: N/A
Computer: NETSERVER
Description:
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: The backup
procedure has been successfully completed.
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 225
Date: 6/9/2009
Time: 12:54:34 AM
User: N/A
Computer: NETSERVER
Description:
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: No log files
can be truncated.
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 223
Date: 6/9/2009
Time: 12:54:17 AM
User: N/A
Computer: NETSERVER
Description:
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: Starting the
backup of log files (range E:\Program Files\Exchsrvr\mdbdata\E000E63B.log -
E:\Program Files\Exchsrvr\mdbdata\E000E63E.log).
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 221
Date: 6/9/2009
Time: 12:54:17 AM
User: N/A
Computer: NETSERVER
Description:
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: Ending the
backup of the file E:\Program Files\Exchsrvr\mdbdata\pub1.stm.
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 222
Date: 6/9/2009
Time: 12:41:43 AM
User: N/A
Computer: NETSERVER
Description:
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: Ending the
backup of the file E:\Program Files\Exchsrvr\mdbdata\priv1.edb. Not all data
in the file has been read (read 11242700800 bytes out of 16215977984 bytes).
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Error
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 217
Date: 6/9/2009
Time: 12:41:43 AM
User: N/A
Computer: NETSERVER
Description:
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: Error (-1018)
during backup of a database (file E:\Program
Files\Exchsrvr\mdbdata\priv1.edb). The database will be unable to restore.
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Error
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 474
Date: 6/9/2009
Time: 12:41:43 AM
User: N/A
Computer: NETSERVER
Description:
Information Store (4628) The database page read from the file "E:\Program
Files\Exchsrvr\mdbdata\priv1.edb" at offset 11242729472 (0x000000029e1e7000)
for 4096 (0x00001000) bytes failed verification due to a page checksum
mismatch. The expected checksum was 4205941953 (0xfab194c1) and the actual
checksum was 571156525 (0x220b282d). The read operation will fail with
error -1018 (0xfffffc06). If this condition persists then please restore
the database from a previous backup.
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Error
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 474
Date: 6/9/2009
Time: 12:41:41 AM
User: N/A
Computer: NETSERVER
Description:
Information Store (4628) The database page read from the file "E:\Program
Files\Exchsrvr\mdbdata\priv1.edb" at offset 11242729472 (0x000000029e1e7000)
for 4096 (0x00001000) bytes failed verification due to a page checksum
mismatch. The expected checksum was 4205941953 (0xfab194c1) and the actual
checksum was 571156525 (0x220b282d). The read operation will fail with
error -1018 (0xfffffc06). If this condition persists then please restore
the database from a previous backup.
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 220
Date: 6/8/2009
Time: 11:40:47 PM
User: N/A
Computer: NETSERVER
Description:
Information Store (4628) 9e13042f-036a-4ed5-9acd-9dbb16f1c4a6: Begining the
backup of the file E:\Program Files\Exchsrvr\mdbdata\priv1.edb (size 15464
Mb).
For more information, click http://www.microsoft.com/contentredirect.asp.