Re: Cluster crashed, need recovery advice

When you "Shut down cluster service" you should also make sure that you
disable the cluster disk driver. You should also make sure that you only
have one node powered on when you're creating the disk. I'd modify your
procedure as follows:

- First make absolutely sure I have a good backup (duh)
- Remove those 2 fail disks from the Cluster Config.
- Shut the cluster service down *** and disable cluster disk driver ***
- Remove the disks from the Windows device manager.
- shutdown this node. startup the other node (without starting the cluster
service there) and remove the 2 disk there as well.
- *** Disable Cluster Disk driver and *** shutdown that node.
- On the SAN: Lowlevel format those 2 disks.
- Startup the first cluster node without enabling the cluster service
- Add the disks back to Windows, format, assign drive-letter
- *** re-enable cluster disk driver then power down first node ***
- Startup the second cluster node without enabling the cluster service
- Add the disks back to Windows, assign drive-letter
- *** re-enable cluster disk driver then shut down this node ***
- Restart the cluster service on node 1and re-attach the 2 disks.
- Restart the cluster service on node 2
- Failover of cluster group 3 to node 2.

Regards,
John

"Tonny van Geloof" <tvNOgeloof@xxxxxxxxxxx> wrote in message
news:hhva61duuh42142c8175cp0hs30opu6mtr@xxxxxxxxxx
> Hello everybody
>
> I've got a serious problem with a W2K cluster.
> I hope there is someone out there who can offer some advice.
>
> I've got a SAN with 11 logical disks defined, which are accessed by a
> cluster of 2 physical nodes.
> 1 disk is quorum, other 10 are data disks.
> 3 cluster groups:
> 1: only quorum drive
> 2: 5 of the datadisks,
> 3: The other 5 datadisks
> Normally node 1 runs cluster group 1+2, and node 2 runs cluster group
> 3 for load balancing purposes.
>
>
> Couple of days ago the Cluster Service itself crashed on node 2
> causing a failiover of cluster group 3 to node 1.
> So far no harm done.
> When I noticed this (it happend overnight) I decided to reboot node 2.
> Assuming everything would come back online, I would just have to do
> a failover on cluster group 3.
> To my horror the Cluster Service on node 1 died when node tried to
> rejoin the cluster.
> All 11 disks were gone.
>
> After a lot of searching I figured out that somehow the administration
> in the "Cluster Disk Device Driver" had gotten screwed.
> Disablling that driver gave me back the disks and all data turned out
> to be save.
> Clearing the "Signatures" key in the registry and restarting
> clusdisk.sys I still have access to those disks, as long as I do that
> on 1 node while the other node is shut down. As soon as the second
> node comes back online all disks immediatly disappear.
> So I can run on one node only.
> When I restart the Cluster Service all everything works but 2 of the
> datadisks remain in fail status in the Cluster Admin, allthough those
> disks show up under "My Computer" and are accessable without any
> problems.
> Eventually I just copied the content of those 2 disks to other
> locations on the disks and I changed the various cluser-shares to
> point to the new locations.
>
> So I'm back online, with limited performance and no redundancy.
>
> I will have to recover somehow...... I came up with the following
> strategy:
>
> - First make absolutely sure I have a good backup (duh)
> - Remove those 2 fail disks from the Cluster Config.
> - Shut the cluster service down
> - Remove the disks from the Windows device manager.
> - shutdown this node. startup the other node (without starting the
> cluster service there) and remove the 2 dissk there as well.
> - shutdown that node.
> - On the SAN: Lowlevel format those 2 disks.
> - Startup the first cluster node without enabling the cluster service
> - Add the disks back to Windows, format, assign drive-letter
> - Startup the second cluster node without enabling the cluster service
> - Add the disks back to Windows, assign drive-letter
> - Restart the cluster service on node 1and re-attach the 2 disks.
> - Restart the cluster service on node 2
> - Failover of cluster group 3 to node 2.
>
> If anything goes wrong again I should be able to get at least back to
> the 1 node cluster config I'm now running on.
>
> Any comments, suggestions, bright ideas, advice are welcome.
>
>
>
>


.

Relevant Pages

  • Daily Report #4165
    ... The resultant cleaned cluster CMDs will ... well-understood host galaxy environment. ... The Nature of the Halos and Thick Disks of Spiral Galaxies ... ACS, NICMOS, and WFPC2 in parallel. ...
    (sci.astro.hubble)
  • Re: Clustering Newbie - SAN Advice
    ... Senior SQL Infrastructure Consultant ... SAN/Smart array or through a fibre channel switch. ... The SAN or Smart array will dictate what internal connection the disks ... single-instance cluster. ...
    (microsoft.public.sqlserver.clustering)
  • changed WWID on cluster member boot disk
    ... single-member cluster; the second member has not yet been added to ... The disks containing the cluster root, ... but an attempt to boot the DS20E as a single-member cluster failed; ... the boot of the stand-alone system, a number of new special device files ...
    (Tru64-UNIX-Managers)
  • Re: Clustering Newbie - SAN Advice
    ... A SAN generally has gigabtes of cache and uses large internal block sizes ... The SAN or Smart array will dictate what internal connection the disks have. ... Senior SQL Infrastructure Consultant ... single-instance cluster. ...
    (microsoft.public.sqlserver.clustering)
  • Re: Cluster Service
    ... MVP - Windows Server - Clustering ... > As long as one node of the cluster is running the cluster service and all ... >> disks owned by another node of the cluster. ...
    (microsoft.public.windows.server.clustering)
Quantcast