Re: OS Question

Well, considering that your software will no doubt be deployed on a single
purpose server and not a desktop PC, using a KM component, nominally a
driver, should not distress you. MS does this for some of their server
products also (ISA, IAS, & Exchange all have, or have had, KM components).



As long as clients access the shared data through an API and not simply via
pointers & offsets, as is suggested by your original question, you should
not have a problem changing the API to use IOCTLs to transfer control to KM
and then do what you do now.


"Hugo gleaves@xxxxxxxxxxx>" <hugh<underbar> wrote in message
news:417E376D-D4DF-4B38-9B1B-B26AE32DD0AA@xxxxxxxxxxxxxxxx


"m" wrote:

Well, you have the answer to your question: The interlocked operation,
implemented as a single instruction, will not execute until your
exception
filter has returned STATUS_CONTINUE_EXECUTION and an arbitrary delay for
thread pre-emption, context switch etc.



As long as you have some mechanism to control access to your pages (call
sequence or locks), this will work just as well as is does on any other
datum. It is not inherently thread safe, but could be made so by VERY
careful implementation. It does not provide a lot of protection, and has
significant overhead, but is not all wrong.



As Maxim points out, the only way to achieve what you really want is to
use
the CPU to enforce the code privilege level and write the code that
modifies
the pages in KM. An IOCTL interface can also be more efficient then
continually changing the page protection in an exception filter.


BTW: I've made the unwarranted assumption that having multiple threads is
directly analogous to having multiple CPUs. As David Craig reminds me,
this
is only true in UM on Windows (because of arbitrary pre-emptive thread
scheduling). It's been too long since I optimized for UP in KM!



"Hugo gleaves@xxxxxxxxxxx>" <hugh<underbar> wrote in message
news:1C0DF457-3490-4C13-98C4-AEF269B76378@xxxxxxxxxxxxxxxx


"David Craig" wrote:

I should have said that in user mode you might be able to keep control
in
one thread by going to realtime, but it is certain it will always
work.
Programming for multiprocessor is always multithread safe, so do it in
a
way
it will be safe for SMP. Many current CPUs are SMP with multiple
cores
and
even the old hyperthreading simulates SMP but it was not quite as good
as
having real multiple cores in a system. There are also tricks with
processor affinity in user mode that can help, but that type of code
can
be
fragile and hard to maintain - so why do it! Use a driver and
spinlocks
to
control access to low level resources. To protect data in user mode
just
use semaphores, mutexes, and/or events as appropriate. Since this is
not
a
user mode code group, it is not really a good question for this
newsgroup
if
that is your target.

"David Craig" <drivers@xxxxxxxxxx> wrote in message
news:ubwXsmItIHA.1316@xxxxxxxxxxxxxxxxxxxxxxx
Also the 'multithreading' can easily be fixed just by raising your
IRQL
to
dispatch level. In user mode you might be able to increase priority
enough to get control. However, this won't fix the multiprocessor
problems. In kernel mode only spinlocks can be used to control
access
to
data or hardware so that only one thread on one processor can have
exclusive control.

"m" <m@xxx> wrote in message
news:et5i0AItIHA.4544@xxxxxxxxxxxxxxxxxxxxxxx
More importantly, the 'protection' that this scheme offers does not
exist
for a period of time when the API call is executing and so any
'rogue'
code can just write to the page then and it will be trashed.



Maxim's situation could be remedied by handling the second
exception
or
using locks on the APIs to prevent concurrent access to pages.


"Maxim S. Shatskih" <maxim@xxxxxxxxxxxxxxxx> wrote in message
news:e%23jB2tHtIHA.4912@xxxxxxxxxxxxxxxxxxxxxxx
Can you explain? what is not compatible with multithreading? If
there
is

- 2 threads wants to write access the same data
- first gain the access
- second thread gains the access
- first thread does its work and returns
- on return, the data is marked back as read-only
- then second thread try to do work, and the data is read-only, a
crash

--
Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
maxim@xxxxxxxxxxxxxxxx
http://www.storagecraft.com








Hi

The problem we face is a real one, and unfortunately Win32 provides
some
exclusion support but not enough. It is not only device drivers that
must
protect resources, we need to protect our own applications shared
structures.

In our case we have an in-memory shared "database" that process
actually
map
into their address space. The memory contains sophisticated structures,
just
like an OS as well as users data.

Obviously an errant thread in some users application could damage some
structure, literally trashing perhaps hundreds of gigs of data
(screwing
up a
list or index tree or something).

However a nag has always been the lack of a fast read/write lock that
we
could use as a spinlock (or something analogous to for very briefly
locked
data).

We now have a pretty decent Read/Write spinlock that is based on
interlock
operations, this provides us with:

1) Far less performance cost than a Mutex (which has no read/write
concept)
2) Permanence (if apps all crash the lock state is retained because
data
is
memory mapped)
3) Upgradability: If a thread has lock in read mode for a time, it can
at
any point request a write lock on same lock, this will (eventually) be
granted.
4) Nesting (Mutexes also provide this)

Thus our API becomes more managable, one function might aquire a read
lock,
then call some other function lower down. This might request a write
lock
on
same lock and so on.

Windows provides us with no such mechanism (unless we dive into kernel
mode
driver stuff) and the cost of Mutexes (even if one did devise some for
of
read/write support) is far greater than an interlock operation.

This has been tested heavily on a dual-core x64 machine, and seem to be
reliable.

The issue that arose a few days ago (or rather a question) was how such
interlock operations will behave if the datum is situated in a
read-only
page
as described earlier in my post.

Finally I agree, this forum is for kernel driver people, BUT I have
tried
in
vain to get expert insights into these issues in other forums and
sadly,
very
very few developers have any real experience or appreciation of the
issues,
driver people always have a superb grasp, all of the comments here for
example betray a firm grasp of the issues, one a standard Win32 forum,
many
would be ill-informed, I have tried.

Thanks for comments though!

Hugh









Hi

Well I'd love to exploit kernel mode as the means of enforcing protection
of
data pages, but this is not something we will be doing in the sort term.

Having said that, is it possible/feasible/good practice to use a kernel
mode
"driver" purely for this purpose?

Bear in mind we want applications to be able to simply "load" the data
into
their address space as is normal for memory mapping.

I have no idea how very large amounts of VM can be accomodated using this
KM
idea, imagine that the system (on 64-bit Windows) could load (say) 100 GB.

It is worth stating that by design, there are system-only pages and
user-pages, that is all system structures appear in one set of mapped
pages,
while only user-data appears in others (imagine that the first 100 pages
are
system strucs, indexes etc and the next 500 pages are pure user data).

Maybe I will get a good book or two, any recommandations?

Thx
Hugh



.

Relevant Pages

  • Re: OS Question
    ... I've made the unwarranted assumption that having multiple threads is ... Use a driver and spinlocks ... To protect data in user mode just ... However a nag has always been the lack of a fast read/write lock that we ...
    (microsoft.public.development.device.drivers)
  • Re: OS Question
    ... Well, you have the answer to your question: The interlocked operation, ... I've made the unwarranted assumption that having multiple threads is ... To protect data in user mode just ... However a nag has always been the lack of a fast read/write lock that we ...
    (microsoft.public.development.device.drivers)
  • Re: OS Question
    ... Programming for multiprocessor is always multithread safe, so do it in a way ... Use a driver and spinlocks to ... To protect data in user mode just ... Permanence (if apps all crash the lock state is retained because data is ...
    (microsoft.public.development.device.drivers)
  • Re: LOR with netisr changes
    ... ::> It's just a bug in the driver. ... the softc and is intended to be covered by the driver's softc lock. ... this is a hack to work around a recursion in net80211. ...
    (freebsd-current)
  • Re: When to write as a device driver instead of just a library.
    ... > one write the code as a device driver as opposed to a plain simple ... In general if it can be done in user mode, ... or major advantages to a kernel mode implementation, ... If you communicate with a device that way, ...
    (comp.os.linux.development.system)