Re: DEP and /NXCOMPAT troubles...

Why don't you just call VirtualProtect with PAGE_EXECUTE_READ?

"Ira Baxter" <idbaxter@xxxxxxxxxxxxxx> wrote in message
news:eq1ektWMJHA.1204@xxxxxxxxxxxxxxxxxxxxxxx
We have long had a robust 32 bit application that loads object code into a
dynamic allocated
buffer and executes it. That lights up Data Execution Prevention if it
is
enabled
when our application is running.

It appears until Vista came along, DEP was enabled by default only for
"essential system programs and services" (according to the buttons
for managing DEP in the control-panel system window).

Because our application (quite reasonably) wasn't an essential
system service, DEP was not enabled when our application ran,
and everything was fine.

But it seems like I'm suddenly getting complaints from Vista users.
Is the default for Vista different? How do I find out except
by opening 100 random Vista systems and inspecting them (ugh)?
(We tested on a Vista system inhouse, but much to our chagrin,
we just figured out that our test Vista box doesn't have the hardware
to support DEP, so it is off on that box no matter what).

Under the assumption that DEP was set for Vista, I have done some
simple experiments, on an XP system that *does* have hardware
support for DEP. Sure enough, when I enable DEP
"for all applications except....", our application is stopped.
When I add our application image path to the exceptions
list, the application runs fine. So, DEP appears to be the problem.

If you dig long enough, you discover a bit (/NOCOMPAT) in the .exe header
which encodes whether the application is "compatible with
Data Execution Prevention" or not. In an attempt to solve
our problem permanently, I tried to set/reset this bit,
using MS VS 2005's EDITBIN with the following command line:
EDITBIN /NXCOMPAT<option> <ourapp.exe>
I tried the option as empty, and as :NO, and after each
attempt, transferred the application to the XP system
with "DEP enabled for all applications" with no exceptions.

The application always failed at the same place.
It appears that EDITBIN isn't doing what I expect.

Any suggestions on how to get this right?
[No, I can't change the application to not execute code].

-- IDB




.

Loading