Re: context switch

On Wed, 04 Jun 2008 00:28:57 -0700, Ryan Liu <rliu-doNotCheck@xxxxxxxxxxxxx> wrote:

I rewote code, and tested on client's enviorment.

I wrote 2 version, one version still uses sync I/O, one server thread per
client. I remove all sleeps, and use same thread write back to client. I
also improved I/O part code. It improves performance. The CPU cost is 0
when serving 87 clients. And momory cost is 30M (My code is small, I think
mostly is .NET enviorment).

I improved sync I/O part and remove sleeps for client program as well. It
also runs faster.

Good. All that sounds just as would be expected. :)

But aftere few hours, client CPU cost up to 100%. I notice all files date
has been changed. I think it is because of virus. I am afraid virus is
taking advantage of new code (no sleep and more efficient I/O), so I add
sleep back to client programs. Now CPU cost is 2-3%. Runs OK. This happend
only on one machine. Other clients runs with client program without sleep,
fine.

Few more hours later, server CPU cost up to 98 too. After restart server,
server runs OK again.

This suggests, obviously, that adding a sleep doesn't help the high CPU issue. Which isn't a surprise.

I don't know why. Maybe it is the virus in network is also taking advantage
of new code which has no sleep and higher I/O effiency.

Absent confirmed knowledge of a virus, as well as some specific information as to how it operates, I would not blame this behavior on a virus. Viruses don't _usually_ "take advantage of" architectural characteristics of random programs. I can't rule it out 100% -- after all, virus writers can be crafty bastards :) -- but it just seems unlikely.

Instead, I think you might actually be looking for a "packrat" bug. That is, some data structure that just keeps adding more and more things to it. Especially if it's a data structure that you regularly scan. Alternatively, it could be a bug where you have some escalating amount of object instantiation. That is, as time goes by, some code that creates multiple objects winds up creating more and more at a time. Even if they are not "packrat"-ed, the cost of allocating and the disposing them may consume excessive CPU time.

Without a concise-but-complete code example to look at, it's not possible to say for sure what this might be. But I'd look for a bug in your own code.

I also notice, even when the server CPU cost is 0, and all clients works
fine, sometime server is very slow to response to Service OnCustomCommand()
which runs on server machine itself.

Again, it's hard to say what might be causing this. Just a random thought: what are the power management settings on the server? Is the server slow to respond only when it's been idle for some time? Windows power management will let a disk spin down, and the time it takes to get the disk back up to speed again can cause delays in operation. Where latency is critical, it's important to make sure the power management settings don't idle the disks.

If that's not the issue, then again without a concise-but-complete code sample, diagnosing it is practically impossible. It could somehow be related to the file locking, but it also might not be.

[...]
I also tried new server with async I/O. Aftere 80 clients connected, server
crashes immediately. In the system event log, I found:

The PowerCatiAppService service terminated unexpectedly. It has done this 7
time(s).
For more information, see Help and Support Center at
http://go.microsoft.com/fwlink/events.asp.

I aslo see applicatoin error in event log:
EventType clr20r3, P1 powercatiappserver.exe, P2 1.0.3076.27877, P3
4844f3ee, P4 system, P5 2.0.0.0, P6 4333ae87, P7 282e, P8 18, P9
system.objectdisposedexception, P10 NIL.

Well, that sure looks like you're trying to access an object that's already been disposed. That would certainly be wrong. :)

Again, without a concise-but-complete code sample, impossible to say why your code is doing that.

Note that I keep mentioning a "concise-but-complete code sample". Of course, it's highly unlikely anyone reading this newsgroup would take the time to navigate your entire server or client code. However, if you can write a very small test application that implements just the basic networking pieces you're trying to get working, that would be useful for consideration.

Note also that it may be useful to _you_ to write such an application even if you never post it. It's a lot easier to learn the specific techniques and to find and fix design and coding errors when you're dealing with a small test application that has no extraneous functionality to distract you.

In other words, you might consider doing your learning on such a small test application, rather than trying to implement the whole thing in-place on production code.

Pete
.

Loading