Re: After manually undoing a recent MS Update, my server is a mess

Susan,

Encouraged by your statement that "[I] didn't get to the underlying cause
though :-( ", I decided to give it one more try. I think I have figured out
the problem, but was hoping you could give me some input on this theory:

BACKGROUND:
Before installing this server, my users were accustomed to storing all of
their files on what they call the "Cases" drive (in a folder labeled with
their names). The "Cases" drive was actually a shared folder on an external
SATA drive (connected by USB to some computer in the workgroup).

When I installed the server, I followed the instructions explicitly and
waited until everything was up and running to make any big changes to the
public shares.

I connected the "Cases" drive to the server and installed an identical drive
to SATA0 inside the server (separate from the two Raid 1 volumes that
contained my OS, Exchange, Admin and Data volumes). I mirrored the two drives
(internal and external) to create a virtual drive (U:\) in which I created a
"users" directory.

I used the SBS2008 Folder Redirection Wizard to redirect everyone's My
Documents folder to the U:\%username% directory. The change happened quickly
because nobody used My Documents to store any files--everything they did was
stored on the "Cases" drive.

Next, I copied all of the user's files from their beloved Cases folder to
their appropriate user folder. I created a public share named "Cases" that
mapped to U:\users\ ... and nobody knew any different. My thought was that by
doing this, if I ever had a problem with the server, I could snag the
external drive and connect it to any workstation immediately for emergency
access to the files (since it was a mirrored volume).

Fortunately, this all worked exceptionally well with one small exception:
Backups. For some reason the SBS2008 Backup software did not want to find the
U: volume while executing. You could assign the drive to be backed up, but
every time it would error out claiming it could not find the volume. I was
never sure if I could safely ignore that message or not and since 98% of the
files used by my office were on the U: drive, I decided I needed to fix it.

During one update, my mirror apparently broke because a new volume appeared
with the same label, but different drive letter. The internal SATA drive was
now U:/ and though it had a lot of the same content as the previous U:/
drive, it was not an exact mirror like I had expected it would be. This was
the final clue that told me something had to be done now.

I setup a D:\Users directory and created a public share for the directory
called "users2" as a temporary measure to redirect user folders. I used the
Folder Redirect Wizard again to move users' folders to the new
"D:\users\%username%" directory and waited for the policy to apply.

Two days later, after an update, I get the BSOD. I think perhaps the problem
was that the update contained something that caused corruption in the drivers
used to read a dynamic volume. This would explain why I could get 3/4 of the
way through the boot process before it went BSOD.

My first reaction, of course, was to restore to a date prior to the update.
I chose to only restore the System since I felt like it was a driver issue,
nothing more. of course, this didn't help because the Restore process worked
for an hour or two and then suddenly restarted the server. I thought maybe it
was done, but BSOD appeared in the same place after restart.

So, proud of my idea to mirror the external drive, I snatched the drive from
the server, went to an unused workstation to mount the drive, share it up and
tell me users to get to their "cases" drive on PC03 instead of the server.
Brilliant, right?
Wrong...Windows XP didn't have the necessary drivers to mount the broken
half-mirror dynamic drive and I was told that because the server was 64 bit,
the 32 bit drivers probably wouldn't even help.

That's when I contacted Dell, then Microsoft and the MS tech helped me to
use the recovery console to manually replace about 10 files embedded deep in
Windows\System32.....

After 4.5 hours, the server finally booted and I was able to get to my
files. The U: volume (previously the mirrored volume) was mounted the I:
volume (the external half of the mirror with the out of date copies of U:/
files) , but the Cases were fine.

For the next couple of days, things appeared to go quite well. That is, of
course, until the Group Policy for Folder Redirection (which I had assigned
on the date that I used for restoral) tried to update the workstations by
telling them to redirect their documents to a share that didn't exist
(because of the restore).

Just tonight I found 3 basic errors on my workstations stating that the
Group Policy could not be applied or that a "service downgrade attack" had
been prevented or that the default storage path for user documents was not
available on-line and the workstation was no longer connected to the network.

Some workstations were fine (if they didn't have very many files to
redirect), but that didn't matter because the server was so slow trying to
resolve all of the incompatable Group Policies, missing shares, broken mirror
volume, etc.

The temporary fix I applied was to delete the scheduled task for Group
Policy deployment (to stop the server from telling the workstations to
redirect to a share that didn't exist) and remove all Folder Redirection
policies (allowing this new policy to propogate quickly). The I restarted
each workstation 3 times (to assure that the policy was applied) and checked
any other drive mappings and/or shortcuts.

This appears to have solved several of the problems, but I am not sure that
it is over. I am just not sure about the state of my server since the OS has
been pieced together in a frankensteinish mixup of backup states and
incompatible settings and the Best Practices tools tell me that everything is
GREAT and that the only error I have can be ignored...

(sorry about the long post, I just figured that perhaps it could help
someone else facing a similar dilemma)

Does this sound reasonable? What should I do from here and what advice do
you have about making sure that something like this never happens again?

Thanks for your patience, your encouragement, and your dedication--I really
appreciate it.

jase



"Susan Bradley" wrote:

Also flash the bios and the nic drivers up to the latest and greatest on
that server before you deploy it.

Susan Bradley wrote:
You didn't get to the underlying cause though :-(

Did you install Win2k8 sp2?
Did you install Drivers?

If you didn't do any of those two things, it could be nothing at all to
do with updating but a hardware/driver/subsystem issue.

You should have exported out the psts to make sure you didn't lose data.

http://www.forensit.com/domain-migration.html
Use that to move the profiles around.

jase wrote:
Susan,

Thanks a lot for all the time you have invested in helping me resolve
this issue.

After evaluating my options, I have decided to wipe the server and
re-install from scratch. It may not be the best method, but it will
undoubtedly take less time and provide more peace of mind than hunting
for some obscure setting I didn't know about.

I am also going to look into a third party backup system that does
bit-wise backups instead of the integrated backup that comes with
SBS2008...it has failed me twice (both times because of some obscure
technicality) and like Bush said, "Fool me once, shame on you, the
fool can't get fooled again": http://www.youtube.com/watch?v=eKgPY1adc0A

If you have any advice about re-installing an SBS2008 server on the
same hardware with around 8 users that only use the file-sharing and
e-mail functions (not power users, really) with focus on reliability
and availability...I'd be glad to hear it.

You have already spent more time on this issue than I intended and
thank you again for all your help.
Best wishes,

Jase

"Susan Bradley" wrote:

Sorry to take so long to get back to you, but I was
checking/comparing to a good box. DTC should be running and is
dependent on RPC.

"About 3/4 of the way through the restore, the
server restarted and did the exact same thing."

This either feels like a bad driver or something in the server is
going bad.

Your best bet would be to hang in there with Microsoft support and
have them keep looking at this.

My only other non drastic suggestion would be to ensure that TOE/RSS
is disabled on the nic cards and to ask you if Windows 2008 SP2 was
installed, and finally what antivirus you are using and if you'd
consider (temporarily) removing it from the server.

jase wrote:
Susan, thanks again for your help.

If IPv6 disabled on that server nic?
No, it's enabled.

Can you see if there is a .dmp file somewhere on the box?
Unfortunately, the debug settings were configured so no .dmp would
be created. I just changed it to Kernel, but that's not gonna help
us right this minute, is it? Sorry about that.

Also, here is the ipconfig/all from one of my workstations.

Windows IP Configuration
Host Name . . . . . . . . . . . . : PC01
Primary Dns Suffix . . . . . . . : ldwolfe.lan
Node Type . . . . . . . . . . . . : Hybrid
IP Routing Enabled. . . . . . . . : No
WINS Proxy Enabled. . . . . . . . : No
DNS Suffix Search List. . . . . . : ldwolfe.lan
ldwolfe.lan
Ethernet adapter Local Area Connection:
Connection-specific DNS Suffix . : ldwolfe.lan
Description . . . . . . . . . . . : Broadcom 440x 10/100
Integrated Controller
Physical Address. . . . . . . . . : 00-1A-A0-04-5D-41
Dhcp Enabled. . . . . . . . . . . : Yes
Autoconfiguration Enabled . . . . : Yes
IP Address. . . . . . . . . . . . : 10.10.10.101
Subnet Mask . . . . . . . . . . . : 255.255.255.0
IP Address. . . . . . . . . . . . : fe80::21a:a0ff:fe04:5d41%4
Default Gateway . . . . . . . . . : 10.10.10.254
DHCP Server . . . . . . . . . . . : 10.10.10.1
DNS Servers . . . . . . . . . . . : 10.10.10.1
fec0:0:0:ffff::1%1
fec0:0:0:ffff::2%1
fec0:0:0:ffff::3%1
Primary WINS Server . . . . . . . : 10.10.10.1
Lease Obtained. . . . . . . . . . : Friday, August 07, 2009
11:54:01 AM
Lease Expires . . . . . . . . . . : Monday, January 18, 2038
11:14:07 PM

Tunnel adapter Teredo Tunneling Pseudo-Interface:
Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : Teredo Tunneling
Pseudo-Interface
Physical Address. . . . . . . . . : FF-FF-FF-FF-FF-FF-FF-FF
Dhcp Enabled. . . . . . . . . . . : No
IP Address. . . . . . . . . . . . : fe80::ffff:ffff:fffd%5
Default Gateway . . . . . . . . . :
NetBIOS over Tcpip. . . . . . . . : Disabled

Tunnel adapter Automatic Tunneling Pseudo-Interface:
Connection-specific DNS Suffix . : ldwolfe.lan
Description . . . . . . . . . . . : Automatic Tunneling
Pseudo-Interface
Physical Address. . . . . . . . . : 0A-0A-0A-65
Dhcp Enabled. . . . . . . . . . . : No
IP Address. . . . . . . . . . . . : fe80::5efe:10.10.10.101%2
Default Gateway . . . . . . . . . :
DNS Servers . . . . . . . . . . . : fec0:0:0:ffff::1%1
fec0:0:0:ffff::2%1
fec0:0:0:ffff::3%1
NetBIOS over Tcpip. . . . . . . . : Disabled

Thanks again for your time.

jase

jase wrote:
Susan, thanks for the prompt reply. Here are the answers to you
questions:

1. Do you know which update as none that I know of disable drivers.
I am simply repeating what I was told by Guarav, the Microsoft
Tech Support agent that helped get my server up and running again.
I asked him how it could get 3/4 of the way through the boot
process, obviously reading files from the drives, before BSOD and
he suggested that perhaps a patch was incompatible with my
hardware. Regardless, even in safe mode, the server would BSOD.
2. Do you have a good backup/system state backup?
Of course the first thing I tried is restoring the system from a
backup image that was two days old. About 3/4 of the way through
the restore, the server restarted and did the exact same thing. I
use SBS 2008 built-in Backup and do a complete backup every day
along with two incremental backups every 24 hours.

3. What's an ipconfig /all from the server and the workstation?
Server IP Configuration

Host Name . . . . . . . . . . . . : CENTRAL
Primary Dns Suffix . . . . . . . : ldwolfe.lan
Node Type . . . . . . . . . . . . : Hybrid
IP Routing Enabled. . . . . . . . : Yes
WINS Proxy Enabled. . . . . . . . : No
DNS Suffix Search List. . . . . . : ldwolfe.lan

Ethernet adapter Local Area Connection:

Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : Broadcom NetXtreme Gigabit
Ethernet
Physical Address. . . . . . . . . : 00-22-19-7B-56-6B
DHCP Enabled. . . . . . . . . . . : No
Autoconfiguration Enabled . . . . : Yes
Link-local IPv6 Address . . . . . :
fe80::6d6c:559f:5639:42b3%10(Preferred)
IPv4 Address. . . . . . . . . . . : 10.10.10.1(Preferred)
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . : 10.10.10.254
DHCPv6 IAID . . . . . . . . . . . : 251666969
DHCPv6 Client DUID. . . . . . . . :
00-01-00-01-10-EE-83-B5-00-22-19-7B-56-6B

DNS Servers . . . . . . . . . . . : 10.10.10.1
Primary WINS Server . . . . . . . : 10.10.10.1
NetBIOS over Tcpip. . . . . . . . : Enabled

Tunnel adapter Local Area Connection* 8:

Media State . . . . . . . . . . . : Media disconnected
Connection-specific DNS Suffix . :
Description . . . . . . . . . . . :
isatap.{E0E21934-0BD5-45E5-9D03-B715F5878
2F3}
Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
DHCP Enabled. . . . . . . . . . . : No
Autoconfiguration Enabled . . . . : Yes


Issues with security patches are a free call.
I was told by the operator that Business Critical After Hours
Support was billed at $190/hr with a two hour minimum.
Fortunately, I had two Support Credits and was able to apply one
to the call.

What was the SRX case on that?
I am not familiar with the term SRX, but the REG number is:
REG:109072169871600
And if you just did one phone call, that's not the price of the
software.
If you consider that 8 attorneys chargin nearly 100/hr were out of
service for 8 hours the first day, I spent 6 hours on the phone
with Dell and 4.5 hours on the phone with Microsoft, you can see
how it adds up and gets costly.
Regardless, my use of hyperbole was more for effect than to claim
that I was unhappy with the service I received. On the contrary, I
was very pleased with Guarav's help and when I contacted him to
re-open the case, he agreed. Unfortunately, he is not available
until Monday, so I came here to ask for your gracious help.

Thanks again for your time and consideration. I will post the
workstation ipconfig/all results in just a minute. Right now I am
logged in remotely.

jase
jase wrote:
Hey all,

I have a small business server 2008 that recently went belly-up
after a "critical update" disabled my RAID and SATA drivers,
giving me a BSOD about 3/4 of the way through the booting
process. The MS Tech Support person helped me boot from CD and
manually strip out the update--it took about 4.5 hours.

Now my server is so slow that it is unusable. I get constant
errors claiming that the Distributive Transaction Coordinator
service shut down or is not available. Restarting the service
doesn't work, it gives an error that says, "if this is a
Microsoft service, try to restart it, if not, contact the vendor".
My users login to the server (with varying degrees of success)
and try to access their files just to find that about every 5
minutes, their cursor locks up (in Word, Outlook, anything) for
about 30 seconds...making it impossible to get anything done.
Even if they save the document to their hard drive to work on
it...same thing. This is happening on all of my Windows XPSP3
Desktops, including two brand-new PCs. I check the error logs
and all I get is the DTC message.

I run the Best Practices App and I get one single error that
doesn't make any sense and I can't resolve: "critical error:
10.10.10.10110.15.10 is not a valid DNS entry" (I'm
paraphrasing). Of course I can't find anything like that in my
DNS records.

I am going to open another trouble ticket today, but at this
point, I have paid twice as much for support (to fix Microsoft's
error) as I did for the entire server. If this wasn't mission
critical, I would have scrapped it and started from scratch.

That's why I was hoping somebody had some information that could
help me. Thanks in advance for your support,

jase

.

Relevant Pages

  • Re: Exchange 2000 Server Very Slow
    ... leak or you are having performance issues with your Disk drives. ... utilities on drives that contain Exchange databases. ... Remove 3rd party software from the server one at a time. ... > That symptom is the exact symptom to a badmail folder issue. ...
    (microsoft.public.exchange2000.general)
  • Re: After manually undoing a recent MS Update, my server is a mess
    ... information to login to my server remotely to look around--hoping he could ... External USB is no replacement for local drives. ... stored on) and chose to only restore the System files. ... The "Cases" drive was actually a shared folder on an external ...
    (microsoft.public.windows.server.sbs)
  • Re: After manually undoing a recent MS Update, my server is a mess
    ... During the restore, the server rebooted without confirmation. ... The "Cases" drive was actually a shared folder on an external ... I mirrored the two drives ...
    (microsoft.public.windows.server.sbs)
  • Re: Email enable doc lib
    ... navigate to the public folder and send some posts with attachments to the ... Microsoft CSS Online Newsgroup Support ... I have disabled forms base Athentication from the default V.Smtp server ...
    (microsoft.public.windows.server.sbs)
  • Re: After manually undoing a recent MS Update, my server is a mess
    ... Keep in mind that I have seen OEM drivers up there so given that you've blown that image away, I can't say for certain that the 'critical' was 100% pushed by Microsoft. ... External USB is no replacement for local drives. ... LOL I show my naivety here when I say that I showed great diligence in my blind application of any MS suggested resolution to "Critical Errors" on my Small Business Server Monitor Report. ... When I got the first BSOD, I booted from the orginal CD and chose to only restore the System files. ...
    (microsoft.public.windows.server.sbs)
Loading