RE: Cluster IP Address Does not fail over

The cluster IP has no dependicies at all. I agree, it should come online with
no problem, but it still fails.

Node1 disk manager sees LUN5. This is my Q: quorum disk. However, it has a
red exclimation point saying "unreadable". Node 2 can read it just fine. Is
this because node2 has control of the shared disks? I would think that if
node1 took over the cluster then, the disks would become readable again in
disk management. This is a production cluster and i cannot bring it offline
to test this.. I wish i could..

Is this how the disk configuration is supposed to work, or is this
'unreadable' part a problem?

DJ

"MarkFox" wrote:

DrumGod,

It sounds like Node1 cannot see the Quorum Disk, has anything changed on
your shared storage, can you verify the Q: LUN is presented properly and you
can see it in Disk Managment. I know it sounds obvious but it Node1 can't
see Q: then it can't bring it online. As for the cluster ip, unless you have
specific dependencies the ip address should come online regardless of the
physical disk. Can you verify that all your NICs are functioning properly,
both Public and Private? Does anything stand out in Cluster Admin as a
problem, under Cluster Configuration - Networks?
--
Mark


"Drumgod" wrote:

All,

I have a 2 node Windows 2003 Server Cluster. Node1 and Node2. Up until now
the failover has been working just fine. Recently we started to have problems
with the cluster Ip address failing over from one node to the other. At this
point i am running the cluster on node2. I CANNOT failover to node1 AT ALL!!!

When i attempt to failover to node1, the cluster ip address status says
FAILED and the quorum disk (q:) stays at status ONLINE PENDING and eventually
fails. This will stay in this state for about 3 minutes and then return to
node2 and the cluster is restored. On node1 the cluster service starts, but
when I failover and the cluster is returned to node2, the cluster service is
then stopped on node1 for some reason. I can then manually restart the
service with no problem. But each time i attempt to failover the cluster
service is stopped on node1.

On node1 I am getting many errors in the event log. Event IDs
1135/1123/1205/57/1069 by source CLUSSVC. Looking up these event does not
lead to a resolution. I have looked at the CLUSTER.LOG file, but im really
not sure what to look for here.

So it seems that when I failover from node2, node1 attempts to take
ownership of the cluster, but the Cluster IP Address fails, then quorum never
comes online and then the cluster returns back to node2.

I have review the Microsoft article about the 'RIGHTS' that the Cluster
service user account must be granted.. thats all correct. All hardware is
working as expected, so I no this is a software issue but do not know where
to look next.

Attached is the last failovers data from my CLUSTER.LOG File on node1

After all my research/troubleshooting I am thinking that DNS or WINS is the
culpret of this issue, am I right on this idea???? I have review all DNS and
WINS settings / records and cannot find anything wrong!!!!

TIA for you help..


Drum on .. .. . . .

DJ A+, Net+, MCP, MCSA




------------------------------Cluster Log Node1 (CASTOR)
-------------------------------
0000053c.00000a38::2006/04/12-12:09:59.732 ERR Physical Disk: [PnP]
GetVolumeInfo: error opening:
\??\Volume{fbaaa819-640d-11da-97d0-806e6f6e6963}, error C0000022.
0000053c.00000a38::2006/04/12-12:09:59.732 WARN Physical Disk: [PnP]
AddVolume: Unable to get volume handle
(\??\Volume{fbaaa819-640d-11da-97d0-806e6f6e6963}), error 5
0000053c.00000a38::2006/04/12-12:09:59.732 INFO Physical Disk: [PnP]
AddVolume: Adding Name
\\?\storage#volume#1&30a96598&0&signaturecf2bda88offset7e00length10f54b3000#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b} - processed
0000053c.00000a38::2006/04/12-12:09:59.732 ERR Physical Disk <Disk Q:>:
Online, volumes not ready. Error: 258.
0000053c.00000a38::2006/04/12-12:09:59.732 INFO Physical Disk <Disk Q:>:
Online, setting ResourceState 4 .
0000053c.00000a38::2006/04/12-12:09:59.732 INFO [RM] RmpSetResourceStatus,
Posting state 4 notification for resource <Disk Q:>
0000053c.00000a38::2006/04/12-12:09:59.732 INFO Physical Disk <Disk Q:>:
Online, returning final error 258 ResourceState 4 Valid 0
00000334.00000ffc::2006/04/12-12:09:59.732 INFO [FM] NotifyCallBackRoutine:
enqueuing event
00000334.00000ffc::2006/04/12-12:09:59.732 INFO [FM] Calling RmNotifyChanges
in monitor 053c.
00000334.00000b78::2006/04/12-12:09:59.732 INFO [FM]
FmpCreateResStateChangeHandler: Entry
00000334.00000b78::2006/04/12-12:09:59.732 INFO [FM]
FmpCreateResStateChangeHandler: Exit, status 0
00000334.00000e90::2006/04/12-12:09:59.732 INFO [FM]
FmpHandleResStateChangeProc: Entry...
00000334.00000e90::2006/04/12-12:09:59.732 INFO [CP] CppResourceNotify for
resource Disk Q:
00000334.00000e90::2006/04/12-12:09:59.732 INFO [DM] DmpQuoObjNotifyCb:
Quorum resource offline/offlinepending/preoffline
00000334.00000e90::2006/04/12-12:09:59.732 WARN [FM]
FmpHandleResourceTransition: Resource Name =
e2276508-cb61-4f9a-9650-5fdd5d5687fc [Disk Q:] old state=129 new state=4
00000334.00000e90::2006/04/12-12:09:59.732 WARN [FM]
FmpHandleResourceTransition: Resource failed, post a work item
00000334.00000e90::2006/04/12-12:09:59.732 INFO [FM]
FmpPropagateResourceState: signalling the ghQuoOnlineEvent
00000334.00000e90::2006/04/12-12:09:59.732 INFO [GUM] GumSendUpdate: queuing
update type 0 context 8
00000334.00000e90::2006/04/12-12:09:59.732 INFO [GUM] GumSendUpdate:
Dispatching seq 286554 type 0 context 8 to node 1
00000334.00000e90::2006/04/12-12:09:59.732 INFO [GUM] GumSendUpdate:
completed update seq 286554 type 0 context 8
00000334.00000e90::2006/04/12-12:09:59.732 INFO [FM]
FmpPropagateResourceState: resource e2276508-cb61-4f9a-9650-5fdd5d5687fc
failed event.
00000334.000008c0::2006/04/12-12:10:00.232 INFO [CP] CppResourceNotify for
resource Cluster IP Address
00000334.000008c0::2006/04/12-12:10:00.232 INFO [FM] FmpRmOnlineResource:
Returning. Resource 15ff5a5e-4eb1-45ad-8b17-51288ee6c513, state 4, status
5027.
00000334.000008c0::2006/04/12-12:10:00.232 INFO [FM] OnlineResource:
dependency 15ff5a5e-4eb1-45ad-8b17-51288ee6c513 failed 0
00000334.00000e90::2006/04/12-12:10:00.232 INFO [FM]
FmpHandleResourceFailure: taking resource
e2276508-cb61-4f9a-9650-5fdd5d5687fc and dependents offline
00000334.00000e90::2006/04/12-12:10:00.232 INFO [FM] FmpHandleGroupFailure,
Entry: Group failure for 2900bc0b-a2e1-4576-a15d-97e09f4d6de7...
00000334.00000e90::2006/04/12-12:10:00.232 INFO [GUM] GumSendUpdate: queuing
update type 0 context 65537
00000334.00000e90::2006/04/12-12:10:00.232 INFO [GUM] GumSendUpdate:
Dispatching seq 286555 type 0 context 65537 to node 1
00000334.00000e90::2006/04/12-12:10:00.232 INFO [GUM] GumSendUpdate:
completed update seq 286555 type 0 context 65537
00000334.00000e90::2006/04/12-12:10:00.232 INFO [DM] DmpQuoObjNotifyCb:
Quorum resource offline/offlinepending/preoffline
00000334.00000e90::2006/04/12-12:10:00.232 INFO [MM] MmSetQuorumOwner(0,0),
old owner 1.
0000053c.00000720::2006/04/12-12:10:00.232 INFO Physical Disk <Disk Q:>:
Terminate, ResourceEntry @ 000A7670 Valid 0
0000053c.00000920::2006/04/12-12:10:00.232 INFO Physical Disk <Disk Q:>:
[PnP] Stop watching PnP events for disk cf2bda88
0000053c.00000920::2006/04/12-12:10:00.232 WARN Physical Disk <Disk Q:>:
[PnP] RemoveDisk: WatchedList is empty
0000053c.00000920::2006/04/12-12:10:00.232 INFO Physical Disk <Disk Q:>:
[PnP] Stop watching disk cf2bda88 - processed
0000053c.00000720::2006/04/12-12:10:00.232 INFO Physical Disk <Disk Q:>:
DiskCleanup started.
0000053c.00000720::2006/04/12-12:10:00.232 INFO Physical Disk <Disk Q:>:
[DiskArb] StopPersistentReservations is called.
0000053c.00000720::2006/04/12-12:10:00.232 INFO Physical Disk <Disk Q:>:
[DiskArb] Stopping reservation thread.
0000053c.000007d8::2006/04/12-12:10:00.232 INFO Physical Disk <Disk Q:>:
[DiskArb] CompletionRoutine, status 0.
0000053c.00000720::2006/04/12-12:10:00.232 INFO Physical Disk <Disk Q:>:
[ArbCleanup] Verifying sector size.
0000053c.00000720::2006/04/12-12:10:00.232 INFO Physical Disk <Disk Q:>:
[ArbCleanup] Reading arbitration block.
0000053c.00000720::2006/04/12-12:10:00.232 INFO Physical Disk <Disk Q:>:
[DiskArb] Successful read (sector 12) [CASTOR:361] (0,c5ca9748:01c65a68).
0000053c.00000720::2006/04/12-12:10:00.232 INFO Physical Disk <Disk Q:>:
[ArbCleanup] Writing arbitration block.
0000053c.00000720::2006/04/12-12:10:00.232 INFO Physical Disk <Disk Q:>:
[DiskArb] Successful write (sector 12) [:0] (0,00000000:00000000).
0000053c.00000720::2006/04/12-12:10:00.232 INFO Physical Disk <Disk Q:>:
[ArbCleanup] Returning status 0.
0000053c.00000720::2006/04/12-12:10:00.232 INFO Physical Disk <Disk Q:>:
[DiskArb] StopPersistentReservations is complete.
0000053c.00000720::2006/04/12-12:10:00.232 INFO Physical Disk <Disk Q:>:
DiskCleanup returning final error 0
00000334.00000e90::2006/04/12-12:10:00.232 INFO [CP] CppResourceNotify for
resource Disk Q:
00000334.00000e90::2006/04/12-12:10:00.232 INFO [DM] DmpQuoObjNotifyCb:
Quorum resource offline/offlinepending/preoffline
00000334.00000e90::2006/04/12-12:10:00.232 INFO [FM] RmTerminateResource:
e2276508-cb61-4f9a-9650-5fdd5d5687fc is now offline
00000334.00000e90::2006/04/12-12:10:00.232 INFO [FM] FmpCleanupQuorum:
Offline resource <Disk Q:> <e2276508-cb61-4f9a-9650-5fdd5d5687fc>
00000334.00000e90::2006/04/12-12:10:00.232 INFO [DM] DmpQuoObjNotifyCb:
Quorum resource offline/offlinepending/preoffline
00000334.00000e90::2006/04/12-12:10:00.232 INFO [DM] DmpQuoObjNotifyCb:
Quorum resource offline/offlinepending/preoffline
00000334.00000e90::2006/04/12-12:10:00.232 INFO [MM] MmSetQuorumOwner(0,0),
old owner 0.
0000053c.00000dac::2006/04/12-12:10:00.232 INFO Physical Disk <Disk Q:>:
Terminate, ResourceEntry @ 000A7670 Valid 0
0000053c.00000920::2006/04/12-12:10:00.232 INFO Physical Disk <Disk Q:>:
[PnP] Stop watching PnP events for disk cf2bda88
0000053c.00000920::2006/04/12-12:10:00.232 WARN Physical Disk <Disk Q:>:
[PnP] RemoveDisk: WatchedList is empty
0000053c.00000920::2006/04/12-12:10:00.232 INFO Physical Disk <Disk Q:>:
[PnP] Stop watching disk cf2bda88 - processed
0000053c.00000dac::2006/04/12-12:10:00.232 INFO Physical Disk <Disk Q:>:
DiskCleanup started.
0000053c.00000dac::2006/04/12-12:10:00.232 INFO Physical Disk <Disk Q:>:
[DiskArb] StopPersistentReservations is called.
0000053c.00000dac::2006/04/12-12:10:00.232 INFO Physical Disk <Disk Q:>:
[DiskArb] StopPersistentReservations is complete.
0000053c.00000dac::2006/04/12-12:10:00.232 INFO Physical Disk <Disk Q:>:
DiskCleanup returning final error 0
00000334.00000e90::2006/04/12-12:10:00.232 INFO [CP] CppResourceNotify for
resource Disk Q:
00000334.00000e90::2006/04/12-12:10:00.232 INFO [DM] DmpQuoObjNotifyCb:
Quorum resource offline/offlinepending/preoffline
00000334.00000e90::2006/04/12-12:10:00.232 INFO [FM] RmTerminateResource:
e2276508-cb61-4f9a-9650-5fdd5d5687fc is now offline
00000334.00000e90::2006/04/12-12:10:00.232 INFO [FM] FmpCleanupQuorum:
RmOfflineResource returns 0
00000334.00000e90::2006/04/12-12:10:00.232 ERR [CS] Halting this node to
prevent an inconsistency within the cluster. Error status = 5027
00000334.000009f8::2006/04/12-12:10:00.232 WARN [ClMsg] Receive datagram
failed, status 995
0000053c.000000fc::2006/04/12-12:10:00.341 WARN [RM] Going away, Status = 1,
Shutdown = 0.
0000053c.000000fc::2006/04/12-12:10:00.341 ERR [RM] Active Resource =
00000000
0000053c.000000fc::2006/04/12-12:10:00.341 ERR [RM] Resource State is 1, ""
0000053c.000000fc::2006/04/12-12:10:00.341 INFO [RM] Posting shutdown
notification.
0000053c.000008d8::2006/04/12-12:10:00.341 INFO [RM] NotifyChanges shutting
down.
0000053c.00000230::2006/04/12-12:10:00.372 INFO [RM] PollerThread stopping.
Shutdown = 1, Status = 258, WaitFailed = 0, NotifyEvent address = 220.
0000053c.000000fc::2006/04/12-12:10:00.841 INFO Physical Disk <Disk V:>:
Terminate, ResourceEntry @ 000A90D0 Valid 0
0000053c.00000920::2006/04/12-12:10:00.841 INFO Physical Disk <Disk V:>:
[PnP] Stop watching PnP events for disk cd9b541e
0000053c.00000920::2006/04/12-12:10:00.841 WARN Physical Disk <Disk V:>:
[PnP] RemoveDisk: WatchedList is empty
0000053c.00000920::2006/04/12-12:10:00.841 INFO Physical Disk <Disk V:>:
[PnP] Stop watching disk cd9b541e - processed
0000053c.000000fc::2006/04/12-12:10:00.841 INFO Physical Disk <Disk V:>:
DiskCleanup started.
0000053c.000000fc::2006/04/12-12:10:00.841 INFO Physical Disk <Disk V:>:
[DiskArb] StopPersistentReservations is called.
0000053c.000000fc::2006/04/12-12:10:00.841 INFO Physical Disk <Disk V:>:
[DiskArb] StopPersistentReservations is complete.
0000053c.000000fc::2006/04/12-12:10:00.841 INFO Physical Disk <Disk V:>:
DiskCleanup returning final error 0
0000053c.000000fc::2006/04/12-12:10:00.841 INFO Physical Disk <Disk V:>:
DisksMountPointCleanup: Cleanup mount point information
0000053c.000000fc::2006/04/12-12:10:00.841 INFO Physical Disk <Disk V:>:
[DiskArb] ArbitrationInfoCleanup.
0000053c.000000fc::2006/04/12-12:10:00.841 INFO Physical Disk <Disk U:>:
Terminate, ResourceEntry @ 000A8A10 Valid 0
0000053c.00000920::2006/04/12-12:10:00.841 INFO Physical Disk <Disk U:>:
[PnP] Stop watching PnP events for disk cd9b541d
0000053c.00000920::2006/04/12-12:10:00.841 WARN Physical Disk <Disk U:>:
[PnP] RemoveDisk: WatchedList is empty
0000053c.00000920::2006/04/12-12:10:00.841 INFO Physical Disk <Disk U:>:
[PnP] Stop watching disk cd9b541d - processed
0000053c.000000fc::2006/04/12-12:10:00.841 INFO Physical Disk <Disk U:>:
DiskCleanup started.
0000053c.000000fc::2006/04/12-12:10:00.841 INFO Physical Disk <Disk U:>:
[DiskArb] StopPersistentReservations is called.
0000053c.000000fc::2006/04/12-12:10:00.841 INFO Physical Disk <Disk U:>:
[DiskArb] StopPersistentReservations is complete.
0000053c.000000fc::2006/04/12-12:10:00.841 INFO Physical Disk <Disk U:>:
DiskCleanup returning final error 0
0000053c.000000fc::2006/04/12-12:10:00.841 INFO Physical Disk <Disk U:>:
DisksMountPointCleanup: Cleanup mount point information
0000053c.000000fc::2006/04/12-12:10:00.841 INFO Physical Disk <Disk U:>:
[DiskArb] ArbitrationInfoCleanup.
0000053c.000000fc::2006/04/12-12:10:00.841 INFO Physical Disk <Disk M:>:
Terminate, ResourceEntry @ 000B4EC0 Valid 0
0000053c.00000920::2006/04/12-12:10:00.841 INFO Physical Disk <Disk M:>:
[PnP] Stop watching PnP events for disk c72c5433
0000053c.00000920::2006/04/12-12:10:00.841 WARN Physical Disk <Disk M:>:
[PnP] RemoveDisk: WatchedList is empty
0000053c.00000920::2006/04/12-12:10:00.841 INFO Physical Disk <Disk M:>:
[PnP] Stop watching disk c72c5433 - processed
0000053c.000000fc::2006/04/12-12:10:00.841 INFO Physical Disk <Disk M:>:
DiskCleanup started.
0000053c.000000fc::2006/04/12-12:10:00.841 INFO Physical Disk <Disk M:>:
[DiskArb] StopPersistentReservations is called.
0000053c.000000fc::2006/04/12-12:10:00.841 INFO Physical Disk <Disk M:>:
[DiskArb] StopPersistentReservations is complete.
0000053c.000000fc::2006/04/12-12:10:00.841 INFO Physical Disk <Disk M:>:
DiskCleanup returning final error 0
0000053c.000000fc::2006/04/12-12:10:00.841 INFO Physical Disk <Disk M:>:
DisksMountPointCleanup: Cleanup mount point information
0000053c.000000fc::2006/04/12-12:10:00.841 INFO Physical Disk <Disk M:>:
[DiskArb] ArbitrationInfoCleanup.
0000053c.000000fc::2006/04/12-12:10:00.841 INFO Physical Disk <Disk D:>:
Terminate, ResourceEntry @ 000B55F8 Valid 0
0000053c.00000920::2006/04/12-12:10:00.841 INFO Physical Disk <Disk D:>:
[PnP] Stop watching PnP events for disk cf2bda87
0000053c.00000920::2006/04/12-12:10:00.841 WARN Physical Disk <Disk D:>:
[PnP] RemoveDisk: WatchedList is empty
0000053c.00000920::2006/04/12-12:10:00.841 INFO Physical Disk <Disk D:>:
[PnP] Stop watching disk cf2bda87 - processed
0000053c.000000fc::2006/04/12-12:10:00.841 INFO Physical Disk <Disk D:>:
DiskCleanup started.
0000053c.000000fc::2006/04/12-12:10:00.841 INFO Physical Disk <Disk D:>:
[DiskArb] StopPersistentReservations is called.
0000053c.000000fc::2006/04/12-12:10:00.841 INFO Physical Disk <Disk D:>:
[DiskArb] StopPersistentReservations is complete.
0000053c.000000fc::2006/04/12-12:10:00.841 INFO Physical Disk <Disk D:>:
DiskCleanup returning final error 0
0000053c.000000fc::2006/04/12-12:10:00.841 INFO Physical Disk <Disk D:>:
DisksMountPointCleanup: Cleanup mount point information
0000053c.000000fc::2006/04/12-12:10:00.841 INFO Physical Disk <Disk D:>:
[DiskArb] ArbitrationInfoCleanup.
0000053c.000000fc::2006/04/12-12:10:00.841 INFO Physical Disk <Disk Q:>:
DisksRelease started, Inserted = 0
.

Relevant Pages

  • RE: Cluster IP Address Does not fail over
    ... outage window for the cluster, shutdown the passive node, then shutdown the ... OK on Node1, then bring up Node2 and test failover, if everything comes ... Node1 disk manager sees LUN5. ... Quorum resource offline/offlinepending/preoffline ...
    (microsoft.public.windows.server.clustering)
  • Re: Cluster log File
    ... Mike I have the problem with cluster disk L: ... resource d6d0ab47-9f75-4666-953c-9ae1e72c6003, error 2. ... [DiskArb] ...
    (microsoft.public.windows.server.clustering)
  • Re: Unable to start the cluster service when creating a new 2 node
    ... existing cluster and could no form a new cluster. ... node 1 as possible host for resource 6add91be-f8ae-4fb7-b78d-2488c502ad0a. ... FmpAddPossibleNodeToList for restype Physical Disk ... [DiskArb] ...
    (microsoft.public.windows.server.clustering)
  • Re: cluster service keeps stoping
    ... called to start cluster. ... Calling FmpAddPossibleNodeToList for restype Physical Disk ... AddVolume: Adding Name ... [DiskArb] ...
    (microsoft.public.windows.server.clustering)
  • Re: Cluster log File
    ... This would have matched the following from the Cluster Log: ... Mike I have the problem with cluster disk L: ... > [DiskArb] StopPersistentReservations is called. ...
    (microsoft.public.windows.server.clustering)