Re: After manually undoing a recent MS Update, my server is a mess

If I were a betting gal, I'd say one of your updates was one of two things:

1. a true driver update (which I never ever ever ever update a driver from Microsoft update). I've been burned too many times, so I just don't do it.

2. Win2k8 sp2 which 'might' have driver updates.

But the fact that you got a bsod during the restore process does not bode well for me. Confirm you did right?

jase wrote:
Susan,

Encouraged by your statement that "[I] didn't get to the underlying cause though :-( ", I decided to give it one more try. I think I have figured out the problem, but was hoping you could give me some input on this theory:

BACKGROUND:
Before installing this server, my users were accustomed to storing all of their files on what they call the "Cases" drive (in a folder labeled with their names). The "Cases" drive was actually a shared folder on an external SATA drive (connected by USB to some computer in the workgroup).

When I installed the server, I followed the instructions explicitly and waited until everything was up and running to make any big changes to the public shares.

I connected the "Cases" drive to the server and installed an identical drive to SATA0 inside the server (separate from the two Raid 1 volumes that contained my OS, Exchange, Admin and Data volumes). I mirrored the two drives (internal and external) to create a virtual drive (U:\) in which I created a "users" directory.

I used the SBS2008 Folder Redirection Wizard to redirect everyone's My Documents folder to the U:\%username% directory. The change happened quickly because nobody used My Documents to store any files--everything they did was stored on the "Cases" drive.

Next, I copied all of the user's files from their beloved Cases folder to their appropriate user folder. I created a public share named "Cases" that mapped to U:\users\ ... and nobody knew any different. My thought was that by doing this, if I ever had a problem with the server, I could snag the external drive and connect it to any workstation immediately for emergency access to the files (since it was a mirrored volume).

Fortunately, this all worked exceptionally well with one small exception: Backups. For some reason the SBS2008 Backup software did not want to find the U: volume while executing. You could assign the drive to be backed up, but every time it would error out claiming it could not find the volume. I was never sure if I could safely ignore that message or not and since 98% of the files used by my office were on the U: drive, I decided I needed to fix it.

During one update, my mirror apparently broke because a new volume appeared with the same label, but different drive letter. The internal SATA drive was now U:/ and though it had a lot of the same content as the previous U:/ drive, it was not an exact mirror like I had expected it would be. This was the final clue that told me something had to be done now.

I setup a D:\Users directory and created a public share for the directory called "users2" as a temporary measure to redirect user folders. I used the Folder Redirect Wizard again to move users' folders to the new "D:\users\%username%" directory and waited for the policy to apply.

Two days later, after an update, I get the BSOD. I think perhaps the problem was that the update contained something that caused corruption in the drivers used to read a dynamic volume. This would explain why I could get 3/4 of the way through the boot process before it went BSOD.

My first reaction, of course, was to restore to a date prior to the update. I chose to only restore the System since I felt like it was a driver issue, nothing more. of course, this didn't help because the Restore process worked for an hour or two and then suddenly restarted the server. I thought maybe it was done, but BSOD appeared in the same place after restart.

So, proud of my idea to mirror the external drive, I snatched the drive from the server, went to an unused workstation to mount the drive, share it up and tell me users to get to their "cases" drive on PC03 instead of the server. Brilliant, right?
Wrong...Windows XP didn't have the necessary drivers to mount the broken half-mirror dynamic drive and I was told that because the server was 64 bit, the 32 bit drivers probably wouldn't even help.

That's when I contacted Dell, then Microsoft and the MS tech helped me to use the recovery console to manually replace about 10 files embedded deep in Windows\System32.....

After 4.5 hours, the server finally booted and I was able to get to my files. The U: volume (previously the mirrored volume) was mounted the I: volume (the external half of the mirror with the out of date copies of U:/ files) , but the Cases were fine.

For the next couple of days, things appeared to go quite well. That is, of course, until the Group Policy for Folder Redirection (which I had assigned on the date that I used for restoral) tried to update the workstations by telling them to redirect their documents to a share that didn't exist (because of the restore).

Just tonight I found 3 basic errors on my workstations stating that the Group Policy could not be applied or that a "service downgrade attack" had been prevented or that the default storage path for user documents was not available on-line and the workstation was no longer connected to the network.

Some workstations were fine (if they didn't have very many files to redirect), but that didn't matter because the server was so slow trying to resolve all of the incompatable Group Policies, missing shares, broken mirror volume, etc.

The temporary fix I applied was to delete the scheduled task for Group Policy deployment (to stop the server from telling the workstations to redirect to a share that didn't exist) and remove all Folder Redirection policies (allowing this new policy to propogate quickly). The I restarted each workstation 3 times (to assure that the policy was applied) and checked any other drive mappings and/or shortcuts.

This appears to have solved several of the problems, but I am not sure that it is over. I am just not sure about the state of my server since the OS has been pieced together in a frankensteinish mixup of backup states and incompatible settings and the Best Practices tools tell me that everything is GREAT and that the only error I have can be ignored...

(sorry about the long post, I just figured that perhaps it could help someone else facing a similar dilemma)

Does this sound reasonable? What should I do from here and what advice do you have about making sure that something like this never happens again?

Thanks for your patience, your encouragement, and your dedication--I really appreciate it.

jase



"Susan Bradley" wrote:

Also flash the bios and the nic drivers up to the latest and greatest on that server before you deploy it.

Susan Bradley wrote:
You didn't get to the underlying cause though :-(

Did you install Win2k8 sp2?
Did you install Drivers?

If you didn't do any of those two things, it could be nothing at all to do with updating but a hardware/driver/subsystem issue.

You should have exported out the psts to make sure you didn't lose data.

http://www.forensit.com/domain-migration.html
Use that to move the profiles around.

jase wrote:
Susan,

Thanks a lot for all the time you have invested in helping me resolve this issue.

After evaluating my options, I have decided to wipe the server and re-install from scratch. It may not be the best method, but it will undoubtedly take less time and provide more peace of mind than hunting for some obscure setting I didn't know about.

I am also going to look into a third party backup system that does bit-wise backups instead of the integrated backup that comes with SBS2008...it has failed me twice (both times because of some obscure technicality) and like Bush said, "Fool me once, shame on you, the fool can't get fooled again": http://www.youtube.com/watch?v=eKgPY1adc0A

If you have any advice about re-installing an SBS2008 server on the same hardware with around 8 users that only use the file-sharing and e-mail functions (not power users, really) with focus on reliability and availability...I'd be glad to hear it.

You have already spent more time on this issue than I intended and thank you again for all your help.
Best wishes,

Jase

"Susan Bradley" wrote:

Sorry to take so long to get back to you, but I was checking/comparing to a good box. DTC should be running and is dependent on RPC.

"About 3/4 of the way through the restore, the
server restarted and did the exact same thing."

This either feels like a bad driver or something in the server is going bad.

Your best bet would be to hang in there with Microsoft support and have them keep looking at this.

My only other non drastic suggestion would be to ensure that TOE/RSS is disabled on the nic cards and to ask you if Windows 2008 SP2 was installed, and finally what antivirus you are using and if you'd consider (temporarily) removing it from the server.

jase wrote:
Susan, thanks again for your help.

If IPv6 disabled on that server nic?
No, it's enabled.

Can you see if there is a .dmp file somewhere on the box?
Unfortunately, the debug settings were configured so no .dmp would be created. I just changed it to Kernel, but that's not gonna help us right this minute, is it? Sorry about that.

Also, here is the ipconfig/all from one of my workstations.

Windows IP Configuration
Host Name . . . . . . . . . . . . : PC01
Primary Dns Suffix . . . . . . . : ldwolfe.lan
Node Type . . . . . . . . . . . . : Hybrid
IP Routing Enabled. . . . . . . . : No
WINS Proxy Enabled. . . . . . . . : No
DNS Suffix Search List. . . . . . : ldwolfe.lan
ldwolfe.lan
Ethernet adapter Local Area Connection:
Connection-specific DNS Suffix . : ldwolfe.lan
Description . . . . . . . . . . . : Broadcom 440x 10/100 Integrated Controller
Physical Address. . . . . . . . . : 00-1A-A0-04-5D-41
Dhcp Enabled. . . . . . . . . . . : Yes
Autoconfiguration Enabled . . . . : Yes
IP Address. . . . . . . . . . . . : 10.10.10.101
Subnet Mask . . . . . . . . . . . : 255.255.255.0
IP Address. . . . . . . . . . . . : fe80::21a:a0ff:fe04:5d41%4
Default Gateway . . . . . . . . . : 10.10.10.254
DHCP Server . . . . . . . . . . . : 10.10.10.1
DNS Servers . . . . . . . . . . . : 10.10.10.1
fec0:0:0:ffff::1%1
fec0:0:0:ffff::2%1
fec0:0:0:ffff::3%1
Primary WINS Server . . . . . . . : 10.10.10.1
Lease Obtained. . . . . . . . . . : Friday, August 07, 2009 11:54:01 AM
Lease Expires . . . . . . . . . . : Monday, January 18, 2038 11:14:07 PM

Tunnel adapter Teredo Tunneling Pseudo-Interface:
Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : Teredo Tunneling Pseudo-Interface
Physical Address. . . . . . . . . : FF-FF-FF-FF-FF-FF-FF-FF
Dhcp Enabled. . . . . . . . . . . : No
IP Address. . . . . . . . . . . . : fe80::ffff:ffff:fffd%5
Default Gateway . . . . . . . . . :
NetBIOS over Tcpip. . . . . . . . : Disabled

Tunnel adapter Automatic Tunneling Pseudo-Interface:
Connection-specific DNS Suffix . : ldwolfe.lan
Description . . . . . . . . . . . : Automatic Tunneling Pseudo-Interface
Physical Address. . . . . . . . . : 0A-0A-0A-65
Dhcp Enabled. . . . . . . . . . . : No
IP Address. . . . . . . . . . . . : fe80::5efe:10.10.10.101%2
Default Gateway . . . . . . . . . :
DNS Servers . . . . . . . . . . . : fec0:0:0:ffff::1%1
fec0:0:0:ffff::2%1
fec0:0:0:ffff::3%1
NetBIOS over Tcpip. . . . . . . . : Disabled

Thanks again for your time.

jase

jase wrote:
Susan, thanks for the prompt reply. Here are the answers to you questions:

1. Do you know which update as none that I know of disable drivers.
I am simply repeating what I was told by Guarav, the Microsoft Tech Support agent that helped get my server up and running again. I asked him how it could get 3/4 of the way through the boot process, obviously reading files from the drives, before BSOD and he suggested that perhaps a patch was incompatible with my hardware. Regardless, even in safe mode, the server would BSOD.
2. Do you have a good backup/system state backup?
Of course the first thing I tried is restoring the system from a backup image that was two days old. About 3/4 of the way through the restore, the server restarted and did the exact same thing. I use SBS 2008 built-in Backup and do a complete backup every day along with two incremental backups every 24 hours.

3. What's an ipconfig /all from the server and the workstation?
Server IP Configuration

Host Name . . . . . . . . . . . . : CENTRAL
Primary Dns Suffix . . . . . . . : ldwolfe.lan
Node Type . . . . . . . . . . . . : Hybrid
IP Routing Enabled. . . . . . . . : Yes
WINS Proxy Enabled. . . . . . . . : No
DNS Suffix Search List. . . . . . : ldwolfe.lan

Ethernet adapter Local Area Connection:

Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : Broadcom NetXtreme Gigabit Ethernet
Physical Address. . . . . . . . . : 00-22-19-7B-56-6B
DHCP Enabled. . . . . . . . . . . : No
Autoconfiguration Enabled . . . . : Yes
Link-local IPv6 Address . . . . . : fe80::6d6c:559f:5639:42b3%10(Preferred)
IPv4 Address. . . . . . . . . . . : 10.10.10.1(Preferred)
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . : 10.10.10.254
DHCPv6 IAID . . . . . . . . . . . : 251666969
DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-10-EE-83-B5-00-22-19-7B-56-6B

DNS Servers . . . . . . . . . . . : 10.10.10.1
Primary WINS Server . . . . . . . : 10.10.10.1
NetBIOS over Tcpip. . . . . . . . : Enabled

Tunnel adapter Local Area Connection* 8:

Media State . . . . . . . . . . . : Media disconnected
Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : isatap.{E0E21934-0BD5-45E5-9D03-B715F5878
2F3}
Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
DHCP Enabled. . . . . . . . . . . : No
Autoconfiguration Enabled . . . . : Yes


Issues with security patches are a free call.
I was told by the operator that Business Critical After Hours Support was billed at $190/hr with a two hour minimum. Fortunately, I had two Support Credits and was able to apply one to the call.

What was the SRX case on that?
I am not familiar with the term SRX, but the REG number is:
REG:109072169871600
And if you just did one phone call, that's not the price of the software.
If you consider that 8 attorneys chargin nearly 100/hr were out of service for 8 hours the first day, I spent 6 hours on the phone with Dell and 4.5 hours on the phone with Microsoft, you can see how it adds up and gets costly.
Regardless, my use of hyperbole was more for effect than to claim that I was unhappy with the service I received. On the contrary, I was very pleased with Guarav's help and when I contacted him to re-open the case, he agreed. Unfortunately, he is not available until Monday, so I came here to ask for your gracious help.

Thanks again for your time and consideration. I will post the workstation ipconfig/all results in just a minute. Right now I am logged in remotely.

jase
jase wrote:
Hey all,

I have a small business server 2008 that recently went belly-up after a "critical update" disabled my RAID and SATA drivers, giving me a BSOD about 3/4 of the way through the booting process. The MS Tech Support person helped me boot from CD and manually strip out the update--it took about 4.5 hours.

Now my server is so slow that it is unusable. I get constant errors claiming that the Distributive Transaction Coordinator service shut down or is not available. Restarting the service doesn't work, it gives an error that says, "if this is a Microsoft service, try to restart it, if not, contact the vendor".
My users login to the server (with varying degrees of success) and try to access their files just to find that about every 5 minutes, their cursor locks up (in Word, Outlook, anything) for about 30 seconds...making it impossible to get anything done. Even if they save the document to their hard drive to work on it...same thing. This is happening on all of my Windows XPSP3 Desktops, including two brand-new PCs. I check the error logs and all I get is the DTC message.

I run the Best Practices App and I get one single error that doesn't make any sense and I can't resolve: "critical error: 10.10.10.10110.15.10 is not a valid DNS entry" (I'm paraphrasing). Of course I can't find anything like that in my DNS records.

I am going to open another trouble ticket today, but at this point, I have paid twice as much for support (to fix Microsoft's error) as I did for the entire server. If this wasn't mission critical, I would have scrapped it and started from scratch.

That's why I was hoping somebody had some information that could help me. Thanks in advance for your support,

jase
.

Relevant Pages

  • Re: After manually undoing a recent MS Update, my server is a mess
    ... Keep in mind that I have seen OEM drivers up there so given that you've blown that image away, I can't say for certain that the 'critical' was 100% pushed by Microsoft. ... External USB is no replacement for local drives. ... LOL I show my naivety here when I say that I showed great diligence in my blind application of any MS suggested resolution to "Critical Errors" on my Small Business Server Monitor Report. ... When I got the first BSOD, I booted from the orginal CD and chose to only restore the System files. ...
    (microsoft.public.windows.server.sbs)
  • Re: Upgrading server motherboard
    ... I used Paragon Drive Copy in the end for the image but I took many drivers ... I found the server board I was supplied had a fault and the customer in the ... relevant drives on your existing server, then restore the images to the new ... can effectively migrate hardware and restore onto the RAID of the new mobo. ...
    (microsoft.public.windows.server.sbs)
  • Re: Changing Hardware Platform
    ... > driver structure then one finds that there are just a few IDE drivers. ... You can't expect PnP to solve your boot controller issue. ... Server 2003 at the moment much. ... >> can be lifted that way unless the controller, drives and Hotswap Chassis ...
    (microsoft.public.windows.server.sbs)
  • Re: After manually undoing a recent MS Update, my server is a mess
    ... Before installing this server, my users were accustomed to storing all of ... The "Cases" drive was actually a shared folder on an external ... I mirrored the two drives ... Connection-specific DNS Suffix. ...
    (microsoft.public.windows.server.sbs)
  • Re: After manually undoing a recent MS Update, my server is a mess
    ... information to login to my server remotely to look around--hoping he could ... External USB is no replacement for local drives. ... stored on) and chose to only restore the System files. ... The "Cases" drive was actually a shared folder on an external ...
    (microsoft.public.windows.server.sbs)
Loading