Re: After manually undoing a recent MS Update, my server is a mess

Keep in mind that I have seen OEM drivers up there so given that you've blown that image away, I can't say for certain that the 'critical' was 100% pushed by Microsoft.

Never install drivers from Microsoft update. It's a rule I live by and haven't regretted it.

When I tried to use a USB for external storage of the shadow copies, it too did not like that and would spontaneously boot the box at random events.

External USB is no replacement for local drives.

jase wrote:
Susan,

Glad to hear you aren't a betting gal. LOL I show my naivety here when I say that I showed great diligence in my blind application of any MS suggested resolution to "Critical Errors" on my Small Business Server Monitor Report. I still find it hard to believe what qualifies as a "critical error" in MS Terms.

Just to clarify: When I got the first BSOD, I booted from the orginal CD (which did not have access to the dynamic volume that my user files were stored on) and chose to only restore the System files. I figured this would replace the corrupt or incompatible driver.

During the restore, the server rebooted without confirmation. I assumed that was normal until the BSOD appeared again. So, I am still not sure if the Server restarted during the restore because of error or because it's supposed to. However, the restore session did not end in BSOD (because it restarted).

Thanks to you, I found another potential problem on my network. I am temporarily using an old router as a gateway. This router is not IPv6 compatible. I will remedy this situation today.

I think the big issue at hand is that it just not wise to try to mirror an external USB drive with an internal SATA drive and use that mirror to store your user documents. Unfortunately, I have to find a way to move all my user files to another disk before I can reclaim the two dynamic disk, which is going to be another pain.

Thanks again for your help, I always learn alot from your posts.

Jase

"Susan Bradley" wrote:

If I were a betting gal, I'd say one of your updates was one of two things:

1. a true driver update (which I never ever ever ever update a driver from Microsoft update). I've been burned too many times, so I just don't do it.

2. Win2k8 sp2 which 'might' have driver updates.

But the fact that you got a bsod during the restore process does not bode well for me. Confirm you did right?

jase wrote:
Susan,

Encouraged by your statement that "[I] didn't get to the underlying cause though :-( ", I decided to give it one more try. I think I have figured out the problem, but was hoping you could give me some input on this theory:

BACKGROUND:
Before installing this server, my users were accustomed to storing all of their files on what they call the "Cases" drive (in a folder labeled with their names). The "Cases" drive was actually a shared folder on an external SATA drive (connected by USB to some computer in the workgroup).

When I installed the server, I followed the instructions explicitly and waited until everything was up and running to make any big changes to the public shares.

I connected the "Cases" drive to the server and installed an identical drive to SATA0 inside the server (separate from the two Raid 1 volumes that contained my OS, Exchange, Admin and Data volumes). I mirrored the two drives (internal and external) to create a virtual drive (U:\) in which I created a "users" directory.

I used the SBS2008 Folder Redirection Wizard to redirect everyone's My Documents folder to the U:\%username% directory. The change happened quickly because nobody used My Documents to store any files--everything they did was stored on the "Cases" drive.

Next, I copied all of the user's files from their beloved Cases folder to their appropriate user folder. I created a public share named "Cases" that mapped to U:\users\ ... and nobody knew any different. My thought was that by doing this, if I ever had a problem with the server, I could snag the external drive and connect it to any workstation immediately for emergency access to the files (since it was a mirrored volume).

Fortunately, this all worked exceptionally well with one small exception: Backups. For some reason the SBS2008 Backup software did not want to find the U: volume while executing. You could assign the drive to be backed up, but every time it would error out claiming it could not find the volume. I was never sure if I could safely ignore that message or not and since 98% of the files used by my office were on the U: drive, I decided I needed to fix it.

During one update, my mirror apparently broke because a new volume appeared with the same label, but different drive letter. The internal SATA drive was now U:/ and though it had a lot of the same content as the previous U:/ drive, it was not an exact mirror like I had expected it would be. This was the final clue that told me something had to be done now.

I setup a D:\Users directory and created a public share for the directory called "users2" as a temporary measure to redirect user folders. I used the Folder Redirect Wizard again to move users' folders to the new "D:\users\%username%" directory and waited for the policy to apply.

Two days later, after an update, I get the BSOD. I think perhaps the problem was that the update contained something that caused corruption in the drivers used to read a dynamic volume. This would explain why I could get 3/4 of the way through the boot process before it went BSOD.

My first reaction, of course, was to restore to a date prior to the update. I chose to only restore the System since I felt like it was a driver issue, nothing more. of course, this didn't help because the Restore process worked for an hour or two and then suddenly restarted the server. I thought maybe it was done, but BSOD appeared in the same place after restart.

So, proud of my idea to mirror the external drive, I snatched the drive from the server, went to an unused workstation to mount the drive, share it up and tell me users to get to their "cases" drive on PC03 instead of the server. Brilliant, right?
Wrong...Windows XP didn't have the necessary drivers to mount the broken half-mirror dynamic drive and I was told that because the server was 64 bit, the 32 bit drivers probably wouldn't even help.

That's when I contacted Dell, then Microsoft and the MS tech helped me to use the recovery console to manually replace about 10 files embedded deep in Windows\System32.....

After 4.5 hours, the server finally booted and I was able to get to my files. The U: volume (previously the mirrored volume) was mounted the I: volume (the external half of the mirror with the out of date copies of U:/ files) , but the Cases were fine.

For the next couple of days, things appeared to go quite well. That is, of course, until the Group Policy for Folder Redirection (which I had assigned on the date that I used for restoral) tried to update the workstations by telling them to redirect their documents to a share that didn't exist (because of the restore).

Just tonight I found 3 basic errors on my workstations stating that the Group Policy could not be applied or that a "service downgrade attack" had been prevented or that the default storage path for user documents was not available on-line and the workstation was no longer connected to the network.

Some workstations were fine (if they didn't have very many files to redirect), but that didn't matter because the server was so slow trying to resolve all of the incompatable Group Policies, missing shares, broken mirror volume, etc.

The temporary fix I applied was to delete the scheduled task for Group Policy deployment (to stop the server from telling the workstations to redirect to a share that didn't exist) and remove all Folder Redirection policies (allowing this new policy to propogate quickly). The I restarted each workstation 3 times (to assure that the policy was applied) and checked any other drive mappings and/or shortcuts.

This appears to have solved several of the problems, but I am not sure that it is over. I am just not sure about the state of my server since the OS has been pieced together in a frankensteinish mixup of backup states and incompatible settings and the Best Practices tools tell me that everything is GREAT and that the only error I have can be ignored...

(sorry about the long post, I just figured that perhaps it could help someone else facing a similar dilemma)

Does this sound reasonable? What should I do from here and what advice do you have about making sure that something like this never happens again?

Thanks for your patience, your encouragement, and your dedication--I really appreciate it.

jase



"Susan Bradley" wrote:

Also flash the bios and the nic drivers up to the latest and greatest on that server before you deploy it.

Susan Bradley wrote:
You didn't get to the underlying cause though :-(

Did you install Win2k8 sp2?
Did you install Drivers?

If you didn't do any of those two things, it could be nothing at all to do with updating but a hardware/driver/subsystem issue.

You should have exported out the psts to make sure you didn't lose data.

http://www.forensit.com/domain-migration.html
Use that to move the profiles around.

jase wrote:
Susan,

Thanks a lot for all the time you have invested in helping me resolve this issue.

After evaluating my options, I have decided to wipe the server and re-install from scratch. It may not be the best method, but it will undoubtedly take less time and provide more peace of mind than hunting for some obscure setting I didn't know about.

I am also going to look into a third party backup system that does bit-wise backups instead of the integrated backup that comes with SBS2008...it has failed me twice (both times because of some obscure technicality) and like Bush said, "Fool me once, shame on you, the fool can't get fooled again": http://www.youtube.com/watch?v=eKgPY1adc0A

If you have any advice about re-installing an SBS2008 server on the same hardware with around 8 users that only use the file-sharing and e-mail functions (not power users, really) with focus on reliability and availability...I'd be glad to hear it.

You have already spent more time on this issue than I intended and thank you again for all your help.
Best wishes,

Jase

"Susan Bradley" wrote:

Sorry to take so long to get back to you, but I was checking/comparing to a good box. DTC should be running and is dependent on RPC.

"About 3/4 of the way through the restore, the
server restarted and did the exact same thing."

This either feels like a bad driver or something in the server is going bad.

Your best bet would be to hang in there with Microsoft support and have them keep looking at this.

My only other non drastic suggestion would be to ensure that TOE/RSS is disabled on the nic cards and to ask you if Windows 2008 SP2 was installed, and finally what antivirus you are using and if you'd consider (temporarily) removing it from the server.

jase wrote:
Susan, thanks again for your help.

If IPv6 disabled on that server nic?
No, it's enabled.

Can you see if there is a .dmp file somewhere on the box?
Unfortunately, the debug settings were configured so no .dmp would be created. I just changed it to Kernel, but that's not gonna help us right this minute, is it? Sorry about that.

Also, here is the ipconfig/all from one of my workstations.

Windows IP Configuration
Host Name . . . . . . . . . . . . : PC01
Primary Dns Suffix . . . . . . . : ldwolfe.lan
Node Type . . . . . . . . . . . . : Hybrid
IP Routing Enabled. . . . . . . . : No
WINS Proxy Enabled. . . . . . . . : No
DNS Suffix Search List. . . . . . : ldwolfe.lan
ldwolfe.lan
Ethernet adapter Local Area Connection:
Connection-specific DNS Suffix . : ldwolfe.lan
Description . . . . . . . . . . . : Broadcom 440x 10/100 Integrated Controller
Physical Address. . . . . . . . . : 00-1A-A0-04-5D-41
Dhcp Enabled. . . . . . . . . . . : Yes
Autoconfiguration Enabled . . . . : Yes
IP Address. . . . . . . . . . . . : 10.10.10.101
Subnet Mask . . . . . . . . . . . : 255.255.255.0
IP Address. . . . . . . . . . . . : fe80::21a:a0ff:fe04:5d41%4
Default Gateway . . . . . . . . . : 10.10.10.254
DHCP Server . . . . . . . . . . . : 10.10.10.1
DNS Servers . . . . . . . . . . . : 10.10.10.1
fec0:0:0:ffff::1%1
fec0:0:0:ffff::2%1
fec0:0:0:ffff::3%1
Primary WINS Server . . . . . . . : 10.10.10.1
Lease Obtained. . . . . . . . . . : Friday, August 07, 2009 11:54:01 AM
Lease Expires . . . . . . . . . . : Monday, January 18, 2038 11:14:07 PM

Tunnel adapter Teredo Tunneling Pseudo-Interface:
Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : Teredo Tunneling Pseudo-Interface
Physical Address. . . . . . . . . : FF-FF-FF-FF-FF-FF-FF-FF
Dhcp Enabled. . . . . . . . . . . : No
IP Address. . . . . . . . . . . . : fe80::ffff:ffff:fffd%5
Default Gateway . . . . . . . . . :
NetBIOS over Tcpip. . . . . . . . : Disabled

Tunnel adapter Automatic Tunneling Pseudo-Interface:
Connection-specific DNS Suffix . : ldwolfe.lan
Description . . . . . . . . . . . : Automatic Tunneling Pseudo-Interface
Physical Address. . . . . . . . . : 0A-0A-0A-65
Dhcp Enabled. . . . . . . . . . . : No
IP Address. . . . . . . . . . . . : fe80::5efe:10.10.10.101%2
Default Gateway . . . . . . . . . :
DNS Servers . . . . . . . . . . . : fec0:0:0:ffff::1%1
fec0:0:0:ffff::2%1
fec0:0:0:ffff::3%1
NetBIOS over Tcpip. . . . . . . . : Disabled

Thanks again for your time.

jase

jase wrote:
Susan, thanks for the prompt reply. Here are the answers to you questions:

1. Do you know which update as none that I know of disable drivers.
I am simply repeating what I was told by Guarav, the Microsoft Tech Support agent that helped get my server up and running again. I asked him how it could get 3/4 of the way through the boot process, obviously reading files from the drives, before BSOD and he suggested that perhaps a patch was incompatible with my hardware. Regardless, even in safe mode, the server would BSOD.
2. Do you have a good backup/system state backup?
Of course the first thing I tried is restoring the system from a backup image that was two days old. About 3/4 of the way through the restore, the server restarted and did the exact same thing. I use SBS 2008 built-in Backup and do a complete backup every day along with two incremental backups every 24 hours.

3. What's an ipconfig /all from the server and the workstation?
Server IP Configuration

Host Name . . . . . . . . . . . . : CENTRAL
Primary Dns Suffix . . . . . . . : ldwolfe.lan
Node Type . . . . . . . . . . . . : Hybrid
IP Routing Enabled. . . . . . . . : Yes
WINS Proxy Enabled. . . . . . . . : No
DNS Suffix Search List. . . . . . : ldwolfe.lan

Ethernet adapter Local Area Connection:
.

Relevant Pages

  • Re: After manually undoing a recent MS Update, my server is a mess
    ... During the restore, the server rebooted without confirmation. ... The "Cases" drive was actually a shared folder on an external ... I mirrored the two drives ...
    (microsoft.public.windows.server.sbs)
  • Re: After manually undoing a recent MS Update, my server is a mess
    ... Before installing this server, my users were accustomed to storing all of their files on what they call the "Cases" drive. ... Wrong...Windows XP didn't have the necessary drivers to mount the broken half-mirror dynamic drive and I was told that because the server was 64 bit, the 32 bit drivers probably wouldn't even help. ... Connection-specific DNS Suffix. ... I asked him how it could get 3/4 of the way through the boot process, obviously reading files from the drives, before BSOD and he suggested that perhaps a patch was incompatible with my hardware. ...
    (microsoft.public.windows.server.sbs)
  • Re: After manually undoing a recent MS Update, my server is a mess
    ... information to login to my server remotely to look around--hoping he could ... External USB is no replacement for local drives. ... stored on) and chose to only restore the System files. ... The "Cases" drive was actually a shared folder on an external ...
    (microsoft.public.windows.server.sbs)
  • Re: Upgrading server motherboard
    ... I used Paragon Drive Copy in the end for the image but I took many drivers ... I found the server board I was supplied had a fault and the customer in the ... relevant drives on your existing server, then restore the images to the new ... can effectively migrate hardware and restore onto the RAID of the new mobo. ...
    (microsoft.public.windows.server.sbs)
  • Re: Changing Hardware Platform
    ... > driver structure then one finds that there are just a few IDE drivers. ... You can't expect PnP to solve your boot controller issue. ... Server 2003 at the moment much. ... >> can be lifted that way unless the controller, drives and Hotswap Chassis ...
    (microsoft.public.windows.server.sbs)