Re: How: multiple program instances sharing same data

Tech-Archive recommends: Repair Windows Errors & Optimize Windows Performance

"Zytan" <zytanlithium@xxxxxxxxx> wrote in message news:1194468230.689513.64710@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Why? How big is the data? Is it really hundreds of megabytes?

The data size is about 10 MB, but the memory used by the process can
be from 6 MB to 90 MB, but usually 25 MB. The program itself is
small, under 1 MB. (And I understand that NET has GC issues, which
makes it hard to see what's being used, and I know there's overhead
for Lists and arrays and strings and objects, and things like that.)

If the data is relatively small but the number of copies is large, then
you have much worse problems than the one you're asking about here.
Running multiple processes can quickly degrade performance, if you're
talking dozens or more.

The number of copies is about 50, times 20 MB each = 1 GB. The
performance is not the problem. It's the memory consumption. I don't
want to use up all the memory for multiple copies of the same data.

Define "usable data". You would have similar issues with memory mapped
files anyway, since a memory mapped file is simply a file mapped to an
otherwise typeless byte array. Depending on the language, you can
usually reference sections of that byte array through typed structures,
but if that's possible then you can do that with a byte array you read
from a file as well.

Right. It looks like if there's no way to share the data as it
resides in memory (in my own C# type) then the only solution is to not
load it all at once, by either explicitly avoiding this, or by using a
memory mapped file, which in both cases means i have to read/convert
the file data to get it into a 'usable' C# type that I can deal with
easily, each time i want to access it. This is ok, I think.

Memory mapped files have two benefits in this scenario, the first being
convenience (you can easily reference a large piece of data as a single
memory object without having to create a whole new swap-file-backed
memory allocation), and the second being the ability to create a memory
mapped file that's really just a shared memory section (i.e. no actual
named file on the disk). Other than that, they behave a lot like just
reading data from an actual file into an array and then accessing that
array via conventional means.

If the file is just an array of text data, and I can access it as a
list of strings from a memory mapped file, and ignoring the little CPU
power required to read and convert it each time, then that's not a bad
idea.

Well, I don't believe that .NET offers any "out of the box" support for
this sort of thing. I don't think any of the options that do exist
should be all that hard to implement, but it's not going to be a
one-liner.

I think there are memory mapped file libraries available, people have
already used pinvoke to access the required win32 functions to do it.

thanks Pete!

Zytan




I don't get it, if you really need to run 50 copies of the same application, just like it's done when running on Terminal Server or Citrix, then you need to keep an eye on your resource usage, that is you need to account for the necessary CPU resources and you need the Memory resources.
In your specific case you say 50 * 20 MB is ~1GB, you are reading data from a common file, but you don't read the file in chunks of 20MB do you? You also said you need to convert the data, that means that this data is not sharable anyway, so your problem is nothing else that the file data right?
If this is true, all you need to do is restrict the amount of data read from the file, there is no need to share this file data through whatever mechanism at all, it just a waste of time to get it right, and it will not solve anything, you still have to account for the converted data which is not sharable
There are another couple of things that aren't clear to me, you said the program is small "under 1MB", well this isn't possible, the smallest managed application consumes at least 8MB of private non sharable memory, so question is what do you mean with this 1MB. Also, you said, the program uses - from 6 MB to 90 MB -, what memory are you talking about here, please note that memory comes as shared an non shared, how much of this is shared? and how comes that memory goes up from 6MB to 90MB, what memory counter are you talking about and how did you measure.

IMO you are looking for a solution for something which isn't a real problem.

Willy.


.

Relevant Pages

  • Re: Array.Resize question
    ... other than low memory ), ... the word resources is used to refer to exactly the things ... array reallocation, and the *vast* majority of classes in the CLR do not ... The statement that Garbage Collection occurs only when memory is low is ...
    (microsoft.public.dotnet.languages.csharp)
  • Re: Fast string operations
    ... Looping: I thought looping over arrays in managed code was "slow" ... array handling and such. ... The problem with TrimHelper is that it always returns a new string instance. ... The customer perceives this as a memory leak. ...
    (microsoft.public.dotnet.languages.csharp)
  • Re: How to point to a global memory block from multiple apps?
    ... > How is MyArrlinked tothe memory mapped file? ... > memory mappoed file to my array memory, ... the mapping address may actually be different in each ...
    (microsoft.public.vb.winapi)
  • Re: High Memory Consumption of Classes and Arrays
    ... Only the array itself has overhead. ... memory as a reference type. ... > least consume 40 bytes of memory. ...
    (microsoft.public.dotnet.framework.performance)
  • Re: Fast linked list
    ... > amounts of memory rather than huge chunks of it. ... random insertions into a vector/dynamic array are not as slow ... to cause a cache miss. ... some hard numbers on speed differences between lists and arrays. ...
    (microsoft.public.vc.language)