Very Strange Network Problem HELP!!!

From: Craig N. (cnash_at_xxplego.com)
Date: 07/16/04 

Date: Fri, 16 Jul 2004 05:36:05 GMT

I have a very strange problem here, been working on it for 4 months now,
hopefully someone can give me some advice.

Anyways, I have a client with 200 users running Citrix. The network
configuration is:

all main servers are HP DL380's, Dual xeon 3.0's, 1 gig of ram
2 domain controllers - win'2000
4 Citrix servers
1 SQL server
1 Print Server
1 EMC Clarian SAN
1 Lotus
1 HP-UX running BPCS
    Switches:
        4 Intel Express Hubs 12 Port
        2 3COM hubs
        2 HP 2524 Switches
        2 Cisco Catalyst 2900 Switches
        1 Cisco Catalyst 3500 Switch
        4 Old SMC Hubs for the phone system

Anyways, the client was running old servers, so they upgraded to HP DL380
servers, and Citrix ran great, then a few days later it went to crap. The
logins were slow, trying to open office documents through office was slow
(but opening through explorer was not), and when typing, it would lag real
bad. (This is Citrix Specific)

I came in to try to fix it, and we tried everything until we finally rebuilt
the domain controllers and Citrix farm, since the problem really seemed like
a server issue. Well, after rebuilding from scratch, we stil had the same
problem, so we tested everything, and it looked good. I then got an idea, it
felt like it might be network traffic.

I took a catalyst 2900 switch that had never been connected to the network,
and ran the servers to it, and 6 thin clients. We turned it on, and the
thing cruised. It rocked, logins were instant, it was the first time it
worked in 4 months. So, I figured I would track down the problem, I started
plugging in switches one at a time until the problem surfaced. I got all the
Intel hubs connected, nothing, then I connected the 3coms, still nothing,
and then the HP switches, so far so good.

By now, its 1:00 AM, and I decide to go home, I quickly connect in the rest
of the switches at once, and BOOM, the problem shows up, so I figure I have
it down to only a few switches, lets go home and try tomorrow. I move the
network cables over to the old servers, then I Connect this switch into the
network, and leave. (It was easier, than moving cables everywhere, since the
switch room is seperate from the server room, and I was tired)

Next Day, I'm excited, so I decide to show it off. WELL, I move my network
cables to the old servers, and unplug the switch from the rest of the
network. The problem is back!!! I reboot the switch, I reboot the servers,
nothing. I even swap network cables. This is the exact same configuration as
last night when it was running great. I dont get it. I guess maybe something
from one cisco switch has infected this one, who knows.

I do know that the HP switches didnt cause a problem, so I empty them out,
and plug in only my servers and thin clients, nothing else, completely
seperate from the network. STILL THE SAME PROBLEM!!

This is weird, and I need to fix it. Its like whenever you add a switch, it
acquires this problem, but a brand new switch runs great until it is
connected to the network for any length of time. It ran great on the other
switch for a good 5 hours, so I know its not the servers, gotta be the
switch. Does anyone have any ideas?

I am resetting my catslyst to factory default then going to try it. Someone
please help, I need this up by monday.

Relevant Pages

  • Re: [fw-wiz] Defense in Depth to the Desktop
    ... > network hardware mechanisms. ... The Strong Internal Network Defense ... The client subnet and the server ... Servers are allowed to reply to clients, ...
    (Firewall-Wizards)
  • [fw-wiz] Defense in Depth to the Desktop
    ... network hardware mechanisms. ... controls is highlighted when the internal network and systems suffer ... The client subnet and the server ... Servers are allowed to reply to clients, ...
    (Firewall-Wizards)
  • 1058 and 1030 errors revisited
    ... tried to run the gpofix utility I got a "network path not ... There are about sixty client ... whereas the servers all have two gigabit ... We printed out our old w2k GPOs and used these to ...
    (microsoft.public.windows.group_policy)
  • Re: [fw-wiz] Defense in Depth to the Desktop
    ... Sounds a lot like Domain Based Security (not Windows 'domains', ... > network hardware mechanisms. ... The client subnet and the ... Servers are allowed to reply to clients, ...
    (Firewall-Wizards)
  • RE: 1058 and 1030 errors revisited
    ... > tried to run the gpofix utility I got a "network path not ... We have four servers to ... There are about sixty client ... > When we bought the machines, ...
    (microsoft.public.windows.group_policy)