Re: Cluster will not fail over.

Comments Inline

"Admiral" <admiral@xxxxxxxxxxxxxxxxxxx> wrote in message
news:%23C%23g6mvCGHA.1088@xxxxxxxxxxxxxxxxxxxxxxx
> Anthony,
>
> I thank you for your response and have gone through the majority of the
> steps you've provided. I also almost applied SP4, but with the type of
> front end application we have that could have been disasterous if not
> tested properly. Considering we are at month end, the powers above have
> blocked me from attempting any further fixes.

Definitely the right idea. Service Packs generally help with stability, but
sometimes they break applications. Testing is important.

>
> I do believe I found the TCP/IP issue/fix, but to be safe it was time to
> call in Microsoft. I gathered all the information I had and have sent it
> their way. I will definitely keep you all posted on what was found and
> solution. I do agree and feel that this clustered environment needs a
> complete overhaul. It has had way too many cooks in the kitchen. For the
> time being, I am covering every avenue by keeping a warm standby along
> with increasing my incrimentals throughout the day.
>
Your experience illustrates exactly where most organizations fall down when
it comes to High Availability. They think a cluster does all the work and
they can ignore the People and Process elements. I have found that strict
procedures that place specific responsibilities on specific individuals are
key to keeping user-generated problems from killing a system. Proper
training and selection of these individuals is also important. Database
operations isn't magic, we need to quit treating it like it is. Develop
procedures that support your operational goals. Make sure the people are
familiar with these procedures and are competent to implement them. Without
that, the best technology in the world is no more reliable than the cheapest
piece of junk.

--
Geoff N. Hiten
Senior Database Administrator
Microsoft SQL Server MVP



> Thanks Again and Happy Holidays!
>
>
> "Anthony Thomas" <ALThomas@xxxxxxxxx> wrote in message
> news:egJ0K5JCGHA.3812@xxxxxxxxxxxxxxxxxxxxxxx
>> Yeah, that's where I was going, you can't even detach "accidentally," or
>> otherwise, the model database unless you are in single user mode or
>> you've
>> started the instance with the 3608 trace flag (by the way, just for
>> grins,
>> there is also 3607 and 3609, which behave the same way, but recover more
>> or
>> fewer databases).
>>
>> As far as the TCP/IP issue goes, you had to rebuild the cluster and were
>> not
>> able to restore the master database. That tells me that you are NOT at
>> the
>> same service pack level as the master backup you were trying to use.
>>
>> Since you went ahead without the master database restored, go ahead and
>> try
>> to reapply the SP3a service pack, or, better yet, apply SP4. If you are
>> using AWE, you will need to apply the post-SP4, 2040 AWE hotfix as well.
>> Also, SP4 breaks the SQLDiag (and it sounds like you will need this); so,
>> you'll want to contact Microsoft Customer Support Services (yes, they've
>> changed their name from MS PSS to MS CSS) to get the private 2148 SQLDiag
>> hotfix.
>>
>> Since you are technically on a brand new cluster (with a lot of old
>> cluster
>> settings laying around), I'd approach this as a fresh build and go
>> through
>> the entire installation configuration just as if nothing was running on
>> it.
>>
>> Geoff mentioned a few things, like comclust for MSDTC and validating the
>> shared clustered disk configuration, but there are several other items
>> after
>> a cluster installation you'll have to revisit. The new installation
>> means
>> that all of the registry settings have reverted back to a fresh
>> installation. The fact you are on a new master database means a lot of
>> the
>> sp_configure settings have reverted back to a fresh installation.
>>
>> You need to check two things, the version of the engine (sqlservr.exe)
>> and
>> the version of the master, model, and msdb databases. You can execute
>> SELECT @@VERSION to get the prior and sp_helpdb master to get the later.
>>
>> Good luck. Keep us up to date with your progress.
>>
>> Sincerely,
>>
>>
>> Anthony Thomas
>>
>>
>> --
>>
>> "john clarke" <jclarke@xxxxxxxxxx> wrote in message
>> news:u0PmezkBGHA.344@xxxxxxxxxxxxxxxxxxxxxxx
>>> ouch...
>>>
>>> this situation happened a while ago for us. one of our techs dettached
>> both
>>> the model AND msdb database (whilst in single user mode) and then sql
>> server
>>> was shut down. Well it would not start up.
>>>
>>> I managed to execute the following command on the server itself:
>>>
>>> sqlservr -c -f -s <instancename> /T3608
>>>
>>> This worked bringing up the sql server in minimal mode. Thereupon i used
>>> query analyser to connect and reattached both the msdb and model
>> databases.
>>>
>>> All was back to normal
>>>
>>> Best of luck!
>>>
>>> john
>>>
>>> "Admiral" <admiral@xxxxxxxxxxxxxxxxxxx> wrote in message
>>> news:uXPvMDkBGHA.628@xxxxxxxxxxxxxxxxxxxxxxx
>>> > Geoff,
>>> >
>>> > I truly appreciate your response. I actually did try to reattach the
>>> > model using this method, unfortunately everytime I would attemp to
>> attach
>>> > the model db, SQL Server would immediately turn off. At the time I
>>> > was
>>> > really pressed for time, which led to the decision for a re-install.
>>> > I
>> do
>>> > believe there is more to the story that was not told to me.
>>> >
>>> > Also, I do apologize if I led you to believe that the cluster is on a
>>> > Windows 2003 platform. It actually resides on a Windows 2000 Advanced
>>> > Server. The link you provided me, should still work for W2k? Like I
>>> > mentioned when it rains it pours and I've been putting out too many
>> fires
>>> > for a Christmas week. I can't thank you enough for the response.
>>> >
>>> >
>>> >
>>> >
>>> > "Geoff N. Hiten" <SQLCraftsman@xxxxxxxxx> wrote in message
>>> > news:OFkKlhbBGHA.1088@xxxxxxxxxxxxxxxxxxxxxxx
>>> > First, what you should have done with a blown model DB:
>>> >
>>> > Start SQL Server in single user mode with trace flag -T3608. This
>>> > stops
>>> > SQL from recovering anything except the master database. Reattach
>> Model.
>>> > If necessary, use files copied from another installation at the exact
>> same
>>> > SP and Hotfix level. Stop SQL Server and restart normally. Sorry,
>>> > but
>> it
>>> > really is that simple. Oh, and lock whoever detached "model" out of
>>> > the
>>> > system. HE is too dangerous to allow near your system.
>>> >
>>> > You didn't mention whether you blew the cluster away or not or just
>>> > rebuilt SQL. If you blew the cluster away, make sure that each disk
>>> > resource has the same drive letter on all nodes and the disk resources
>>> > fail over correctly from node to node. Stop the resource group, move
>> it,
>>> > and start each disk resource independently to test.
>>> >
>>> > The Named Pipes only issue sounds like an incomplete SP3a install.
>>> > Windows 2003 will prevent TCP/IP access if it detects a pre-SP3a SQL
>>> > installation. Follow this article and re-apply SP3a.
>>> >
>>> > http://support.microsoft.com/default.aspx?scid=kb;en-us;815431
>>> >
>>> >
>>> > --
>>> > Geoff N. Hiten
>>> > Senior Database Administrator
>>> > Microsoft SQL Server MVP
>>> >
>>> >
>>> >
>>> >
>>> > "Admiral" <admiral@xxxxxxxxxxxxxxxxxxx> wrote in message
>>> > news:uoXFWWbBGHA.216@xxxxxxxxxxxxxxxxxxxxxxx
>>> > We had an error over the weekend of mass porportions(Sunday 3pm PST).
>>> > Long story short; the model database was detached and the SQL Server
>>> > was
>>> > stopped, with it still detached. This happened to happen on our
>>> > primary
>>> > Production Database Clustered Server which is the bread-n-butter of
>>> > the
>>> > compay. (OUCH!)
>>> >
>>> > It was time for some fast actions. We started the re-install SQL
>> Server.
>>> > In order to do so, the previous install had to be uninstalled. This
>>> > seemed to go smoothly enough, but when re-applying the SP3a, we
>>> > encountered an error. After researching the error, apparently in a
>>> > clustered environment this will occur since the SP3a files still
>>> > reside
>> on
>>> > the node(s). Microsoft states that if within a particular log file it
>>> > results with an 'Installation was Successful', to disregard the error.
>> I
>>> > double checked the log file and sure enough the error was disregarded.
>>> >
>>> > We moved along with the installation. We were able to restore all
>>> > the
>>> > user databases and all system databases with the exception of the
>>> > master
>>> > database. Unfortunately, even with starting SQL Server in single-user
>>> > mode, the restore of the master database would not take. So it was
>>> > not
>>> > restored, but all other databases were. Fortunately, I ran a quick
>> script
>>> > to recover all the user logins previous to the disaster, which I
>> reapplied
>>> > to the new installation of SQL Server. Everything came back up and
>>> > the
>> QA
>>> > Team successfully tested the production Application (Monday 4am PST).
>>> > (Fhweeh)
>>> >
>>> > After the succesful testing of the production environment, we tested
>> the
>>> > fail-over which resulted in SQL Server not starting on the secondary
>> node.
>>> > All the resources came right up on it, but not SQL Server. The only
>> error
>>> > that was that it was not able to locate the file on
>> 'O\logs\mastlog.ldf'.
>>> > This error did not make sense since SQL Server uses the same file for
>> the
>>> > primary node. We were pressed for time since it was closing to start
>>> > of
>>> > business East Coast time, so we left the server as is.
>>> >
>>> > Throughout the day there were other issues that arose, one in
>> particular
>>> > was certain systems were not able to connect to the server via TCP/IP.
>> In
>>> > order to have them connect they needed to create an alias of the
>>> > server
>>> > and use Name Pipes. This seems to be a rising concern because there
>>> > are
>>> > users who need to connect via ODBC to a widely used particular Access
>>> > Application, which seems to only like the TCP/IP route. I am somewhat
>>> > sure this is related to the cluster failure.
>>> >
>>> > Anyway, this is the first time I've had to take a breathe to revisit
>> the
>>> > problem at hand. We have been dealing with another server that
>>> > crashed
>> on
>>> > the same day, resulting in a brand new build of a SQL Server Cluster
>>> > environment (completely non related to the issue at hand).
>>> >
>>> > I'm sorry for the long winded story. Would you have any idea as to
>>> > why
>>> > the cluster would fail on failover along with the TCP/IP issue?
>>> >
>>> > Thanks in Advanced..
>>> >
>>> >
>>> >
>>> >
>>> >
>>>
>>>
>>
>>
>
>


.

Loading