Re: Reprise: Pended IOCtrl + LoadLibrary == Deadlock?

Dear Ivan,

> > You mention below turning on handle tracing in devenv.exe. After
poking
> > around for a bit I have realized that this will not work for me as I
am
> > running DevStu 6, not .NET.
> >
> handle tracing is an OS concept, and it does not depend on the target
> application
> or on the debugger. Try this:
> c:\>gflags /v IMAGE /enable devenv.exe HANDLE_CHECKS
> c:\>gflags /v IMAGE /enable TestApp.exe HANDLE_CHECKS

I see, I did not understand completely what you were asking before. I
now get it (and see that using gflags is the same as using appverifier
insofar as the -v option is concerned. It obviously has other nice
features as well).



> and then run devenv.exe under cdb/ntsd/windbg, and, when you have
> the original probelm in the debugger, attach a non invasive debugger
> to the target application spawned by devenv.exe (cdb -pv -p <PID>)
> and execute !htrace in both devenv.exe and the target app.

Ok, I did this and have included more discussion and my results at the
bottom of this post.



> > When I run the app (F5) I get fault traps that I had never seen
before
> > (presumably generated by AV).
>
> You should prompthly investigate those new AVs, because they may show
> that the root cause of the problem is yet to be found

You are a taskmaster! :) I have included this case in the results
below.


> > If I run the app either directly from the shell or via windbug
> > _on_the_target_ (not from my km debugging host I have been
mentioning
> > before) the app does not fail and naturally AV and !htrace confirm
that
> > nothing odd is happening. The handle that the driver ends up with
is
> > used several times for other purposes before the driver gets the
value,
> > but they are all perfectly symetrical open/close pairs and no events
on
> > that handle value occur after the driver is opened.
> >
> You should pursue the root cause from the scenario when the app
> is launched from devenv.exe. It's most likely an interaction problem
with
> devenv.exe


It is looking that way, which is some ways is a relief as my customers
will not be doing this.




*******************************************************
More discussion
*******************************************************
Ok, I rebuilt everything the way I was before so that I could reproduce
the bug again. I then tried all possible compinations of the following
three things:

1) running my app with and without handle tracking
2) running msdev.exe with and without handle tracking
3) launching msdev.exe from the shell and from within windbg.

The bottom line in that the only way I can reproduce the crash is when
all the handle tracking is off and I am running msdev from the shell -
i.e. the way I was doing things when I first saw this bug 3 weeks ago.

All other cases either result in not being able to run msdev or my app
at all, or with no troubles.

Unfortunately I cannot tell if this is a bug in my app revealed by
msdev, or a bug in msdev revealed by my app. I am actually leaning
towards the latter as I am seeing a number of handle errors in msdev
itself both when it starts up and when it is trying to load my app. I
am also seeing errors in my apps CRT init (which is completely out of my
hands).

I am still going to try Alexanders suggestion of trying to read the
startupinfo in my app when I first get control and see if that is in any
way telling. That may give me enough information to sort the mystery
once and for all.

One happy upshot of everything is that I have discovered that I can run
my app in its original build configuration without seeing the bug if I
load msdev from within windbg. I can even have handle tracking enabled
for both my app and msdev. So while I cannot debug the problem at the
very moment, if it is infact an issue with my app (rather than with
msdev), it may well show up again later in a different manifestation.
If it does, I will at least be in a position to bring more tools to bear
upon it.

In the meantime, I can move forward on this project. I have been
working on this bug for nearly 3 weeks now and am sorely behind
schedule. I simply cannot spend any more time on it in its current
state (well after trying Alexander's suggestion of course).


Best regards,

Tom




*******************************************************
Results of my testing
*******************************************************

av+ means appverifier/gflags handle tracking only enabled.
av- means appverifier/gflags disabled

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
run msdev.exe(av-)
load (from within msdev) ionwizrd.exe(av-)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
lockup occurs.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
run windbg
load (from within windbg) msdev.exe(av-)
load (from within msdev) ionwizrd.exe(av-)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
no lockup.

But, strangely I get a C0000008 in the process of loading
msdev. Output of windbg is:
Microsoft (R) Windows Debugger Version 6.5.0003.7
Copyright (c) Microsoft Corporation. All rights reserved.

CommandLine: I:\devstu\MSDev98\Bin\MSDEV.EXE
Symbol search path is: <..snip..>
Executable search path is: <..snip..>
ModLoad: 00400000 00443000 msdevP.exe
ModLoad: 77f50000 77ff7000 ntdll.dll
ModLoad: 77e60000 77f46000 D:\WINDOWS\system32\kernel32.dll
ModLoad: 73dd0000 73ec2000 D:\WINDOWS\System32\MFC42.DLL
ModLoad: 77c10000 77c63000 D:\WINDOWS\system32\MSVCRT.dll
ModLoad: 7e090000 7e0d1000 D:\WINDOWS\system32\GDI32.dll
ModLoad: 77d40000 77dcc000 D:\WINDOWS\system32\USER32.dll
ModLoad: 77dd0000 77e5d000 D:\WINDOWS\system32\ADVAPI32.dll
ModLoad: 78000000 78087000 D:\WINDOWS\system32\RPCRT4.dll
ModLoad: 50000000 50146000 I:\devstu\MSDev98\Bin\devshl.dll
ModLoad: 771b0000 772d4000 D:\WINDOWS\system32\ole32.dll
ModLoad: 77120000 771ab000 D:\WINDOWS\system32\OLEAUT32.dll
ModLoad: 773d0000 77bca000 D:\WINDOWS\system32\SHELL32.dll
ModLoad: 70a70000 70ad4000 D:\WINDOWS\system32\SHLWAPI.dll
ModLoad: 77340000 773cb000 D:\WINDOWS\system32\COMCTL32.dll
(6a4.6a8): Break instruction exception - code 80000003 (first chance)
eax=00251ea4 ebx=7ffdf000 ecx=00000003 edx=77f51308 esi=00251ea4
edi=00251f18
eip=77f75a58 esp=0013fb38 ebp=0013fc2c iopl=0 nv up ei pl nz na
pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000
efl=00000202
ntdll!DbgBreakPoint:
77f75a58 cc int 3

The stack is:

Args to Child
7ffde000 7ffdf000 00000000 ntdll!DbgBreakPoint (FPO: [0,0,0])
0013fd30 77e60000 0013fce8 ntdll!LdrpInitializeProcess+0xdf3 (FPO:
[5,82,3])
0013fd30 77f50000 00000000 ntdll!LdrpInitialize+0x186 (FPO: [Non-Fpo])
00000000 00000000 00000000 ntdll!KiUserApcDispatcher+0x7

Other than this, there are no warnings or errors.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
run msdev.exe(av-)
load (from within msdev) ionwizrd.exe(av+)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

msdev loads fine. When I hit F5 to run my app the following occurs:

exception c0000008 (this I get via a dialog box from msdev)

stack is (note this is before my apps CRT init):

NTDLL! 77f75e2f()
7ffe0304()
MSVCRT! 77c2b4b1()
NTDLL! 77f5b42c()
NTDLL! 77f56771()
NTDLL! 77f649a8()
00020000()

Hit F5, say do not pass on to app being debugged.

exception c0000008 (now we are at my apps CRT init. This looks very
similar to the
stack when I lock up loading the monoatd4 dll later)

NTDLL! 77f75e2f()
7ffe0304()
_ioinit() line 250 + 60 bytes
WinMainCRTStartup() line 169
KERNEL32! 77e814c7()


Hit F5, say do not pass on to app being debugged.

exception c0000008 (now we loading one of my plugins. Again in CRT init
and again this looks very similar to the
stack when I lock up loading the monoatd4 dll later)

NTDLL! 77f75e2f()
7ffe0304()
_ioinit() line 250 + 60 bytes
_CRT_INIT(void * 0x018e0000, unsigned long 0x00000001, void *
0x00000000) line 127
_DllMainCRTStartup(void * 0x018e0000, unsigned long 0x00000001, void *
0x00000000) line 231 + 17 bytes
NTDLL! 77f5b42c()
NTDLL! 77f56771()
NTDLL! 77f6151e()
NTDLL! 77f570e0()
KERNEL32! 77e7d901()
KERNEL32! 77e7d95e()
SystemDLL::SystemDLL(const STRINGc & {...}, int 0x00000001) line 96 + 18
bytes
DynamicLibInst::Create(const STRINGc & {...}, int 0x00000001) line 222 +
35 bytes

This continues for every DLL that is loaded. I gave up after about 10.


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
run windbg
load (from within windbg) msdev.exe(av-);
load (from within msdev) ionwizrd.exe(av+)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Same break on the load of msdev as before. No other troubles.


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
load msdev.exe(av+)
load (from within msdev) ionwizrd.exe(av+)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The app terimates with a message saying "The application failed to
initialize properly (0xc0000008).
Click on OK to terminate the application." If I click "OK" I get the
same message a
second time and then indeed the app terminates.

I presume this is the same c0000008 that I am seeing when I run
msdev.exe from within windbg.



~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
run windbg
load (from within windbg) msdev.exe(av+)
load (from within msdev) ionwizrd.exe(av+)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Same break on the load of msdev as before. No other troubles.









.

Relevant Pages

  • Re: Reprise: Pended IOCtrl + LoadLibrary == Deadlock?
    ... >3) launching msdev.exe from the shell and from within windbg. ... >load (from within msdev) ionwizrd.exe ... When I hit F5 to run my app the following occurs: ...
    (microsoft.public.development.device.drivers)
  • Re: Reprise: Pended IOCtrl + LoadLibrary == Deadlock?
    ... I cannot run msdev with tracing on. ... > load ionwizrd.exe ... > second time and then indeed the app terminates. ... > when your target process is in this configuration, ...
    (microsoft.public.development.device.drivers)
  • Re: Reprise: Pended IOCtrl + LoadLibrary == Deadlock?
    ... Yes it looks like your deadlock is also caused by this shell issue ... > ran my app from msdev. ... > to load with C0000008 error. ...
    (microsoft.public.development.device.drivers)
  • Re: Reprise: Pended IOCtrl + LoadLibrary == Deadlock?
    ... The original trouble was a deadlock in my app that turned out to be ... Ivan suggested that I gflag both my app and msdev to HANDLE_CHECKS. ... It was observed that msdev would not load at all in this state - fails ... used windbg to load msdev in an attempt to "debug the debugger" the ...
    (microsoft.public.development.device.drivers)
  • Re: Mobile Development for WM 5 with one main exe controlling mult
    ... I'm trying to convince them with what you said about the load strategy, ... it could be due to the fact that all the loading are done ... App size has nothing ... split the whole program from one exe to multiple. ...
    (microsoft.public.pocketpc.developer)
Loading