"Attempt to execute non-executable address" on Server 2008 when Terminal Services installed

[Corinna Vinschen]

Hi,

I'm not sure if there's another, better suited managed newsgroup for
this kind of problem. I opened a support case at Microsoft Professional
Support yesterday, but they refused to take it because the actual
problem (the crash) doesn't occur in a Windows component. The support
engineer suggested to ask in one of the managed newsgroups, so here we
are.

Months ago I tweaked Cygwin so that it works on Windows Server 2008.
There was a single problem with SEH which has been discussed in this
newsgroup back then. After this has been fixed, Cygwin worked nicely on
Server 2008. Once in while a Cygwin user claimed that Cygwin's bash
crashes on their 2008 machines, but since I was never able to reproduce
the problem, I doubted that this had anything to do with Cygwin itself.

Last week a Cygwin user noted that Cygwin's bash crashes as soon as
Terminal Services have been installed on Server 2008. I tested this
observation and could reproduce it. Bash, as well as some other
applications (GDB, grep) crash, but most other applications still run
fine. I started to debug this phenomenon and what happens is this:

The crash occurs in plain application code, not in the Cygwin DLL. The
application's main() function calls some arbitrary other application
function and the first assembler instruction in this called function
crashes with a STATUS_ACCESS_VIOLATION. Analyzing this exception with
WinDbg shows that the reason for the SEGV is an "Attempt to execute
non-executable address XYZ", with XYZ the address of the first (and
crashing) assembler instruction in the called function.

The address in question is well within the .text segment of the
executable. There's no visible reason why this address should be a
non-executable address.

As it turned out, this weird crash only occurs when the application is
running with DEP switched on (default on Server 2008) *and* Terminal
Services are installed. If you switch off DEP, or you deinstall
Terminal Services, the crash doesn't occur. It doesn't matter whether
Terminal Services are running or not, just the fact that they are
installed is sufficient to trigger the crash. You can deinstall and
reinstall TS multiple times and the behaviour is reproducible, no crash
without TS, crash with TS. It also does not matter whether you install
Cygwin before TS is installed or afterwards. Cygwin has no dependencies
to special TS properties anyway.

For testing purposes I switched off parts of the Cygwin initialization
code one at a time, for instance installing SEH handler, Ctrl-C handler,
stuff like that, but to no avail.

So, now I'm at a loss what to look for. I have a hacky workaround which
is to switch off DEP in the Cygwin initialization code, but that's
really just a hack, IMO. Of course that only works if DEP is running in
OptOut mode.

Any help, *especially* (but not only) from Microsoft, would be really
appreciated.


Corinna

[Pavel A.]

Corinna,

You wrote that the crash occurs immediately when main() calls
out to any other function, correct?
Can it be that something occurs in the init code
that (for example) causes stack corruption, or changes DEP status of the
app memory?
If these apps written in c++, constructors of static objects may run
just before main() and call some other lib stuff?

Regards,
--PA

Post by Corinna Vinschen
Hi,
I'm not sure if there's another, better suited managed newsgroup for
this kind of problem. I opened a support case at Microsoft Professional
Support yesterday, but they refused to take it because the actual
problem (the crash) doesn't occur in a Windows component. The support
engineer suggested to ask in one of the managed newsgroups, so here we
are.
Months ago I tweaked Cygwin so that it works on Windows Server 2008.
There was a single problem with SEH which has been discussed in this
newsgroup back then. After this has been fixed, Cygwin worked nicely on
Server 2008. Once in while a Cygwin user claimed that Cygwin's bash
crashes on their 2008 machines, but since I was never able to reproduce
the problem, I doubted that this had anything to do with Cygwin itself.
Last week a Cygwin user noted that Cygwin's bash crashes as soon as
Terminal Services have been installed on Server 2008. I tested this
observation and could reproduce it. Bash, as well as some other
applications (GDB, grep) crash, but most other applications still run
The crash occurs in plain application code, not in the Cygwin DLL. The
application's main() function calls some arbitrary other application
function and the first assembler instruction in this called function
crashes with a STATUS_ACCESS_VIOLATION. Analyzing this exception with
WinDbg shows that the reason for the SEGV is an "Attempt to execute
non-executable address XYZ", with XYZ the address of the first (and
crashing) assembler instruction in the called function.
The address in question is well within the .text segment of the
executable. There's no visible reason why this address should be a
non-executable address.
As it turned out, this weird crash only occurs when the application is
running with DEP switched on (default on Server 2008) *and* Terminal
Services are installed. If you switch off DEP, or you deinstall
Terminal Services, the crash doesn't occur. It doesn't matter whether
Terminal Services are running or not, just the fact that they are
installed is sufficient to trigger the crash. You can deinstall and
reinstall TS multiple times and the behaviour is reproducible, no crash
without TS, crash with TS. It also does not matter whether you install
Cygwin before TS is installed or afterwards. Cygwin has no dependencies
to special TS properties anyway.
For testing purposes I switched off parts of the Cygwin initialization
code one at a time, for instance installing SEH handler, Ctrl-C handler,
stuff like that, but to no avail.
So, now I'm at a loss what to look for. I have a hacky workaround which
is to switch off DEP in the Cygwin initialization code, but that's
really just a hack, IMO. Of course that only works if DEP is running in
OptOut mode.
Any help, *especially* (but not only) from Microsoft, would be really
appreciated.
Corinna

[Corinna Vinschen]

Hi Pavel,

Post by Pavel A.
Corinna,
You wrote that the crash occurs immediately when main() calls
out to any other function, correct?

Yes, exactly the first assembler instruction in the child function
crashes.

Post by Pavel A.
Can it be that something occurs in the init code
that (for example) causes stack corruption, or changes DEP status of the
app memory?

There's never any guarantee that there's no bug in the init code, but
there's at least a 99.9% certainty that the DEP status isn't changed in
the init code. The fact that Cygwin runs fine with DEP on a non-TS 2008
is not exactly helping here...

Post by Pavel A.
If these apps written in c++, constructors of static objects may run
just before main() and call some other lib stuff?

Personally I only found three crashing apps so far (bash, gdb, grep) and
all of them are written in plain C. What runs before main is the Cygwin
process initialization code. The main() function is called from there.
However, I examined the stack before (after the call instruction in
main, but before the push %ebp instruction in the child function) and
after the crash. The stack looks sane. The register content
(especially ebp and esp) looks ok as well.


Thanks,
Corinna

[r***@gmail.com]

Post by Corinna Vinschen
Hi Pavel,

Post by Pavel A.
Corinna,
You wrote that the crash occurs immediately when main() calls
out to any other function, correct?

Yes, exactly the first assembler instruction in the child function
crashes.

Post by Pavel A.
Can it be that something occurs in the init code
that (for example) causes stack corruption, or changes DEP status of the
app memory?

There's never any guarantee that there's no bug in the init code, but
there's at least a 99.9% certainty that the DEP status isn't changed in
the init code.

Does Cygwin initialisation use VirtualProtect() at any point?
I had a similar sounding bug in the past which turned out to be caused
by some self-modifying code that wasn't correctly restoring the
PAGE_EXECUTE bits.

Regards,
Roger.

 The fact that Cygwin runs fine with DEP on a non-TS 2008

Post by Corinna Vinschen
is not exactly helping here...

Post by Pavel A.
If these apps written in c++, constructors of static objects may run
just before main() and call some other lib stuff?

Personally I only found three crashing apps so far (bash, gdb, grep) and
all of them are written in plain C.  What runs before main is the Cygwin
process initialization code.  The main() function is called from there.
However, I examined the stack before (after the call instruction in
main, but before the push %ebp instruction in the child function) and
after the crash.  The stack looks sane.  The register content
(especially ebp and esp) looks ok as well.
Thanks,
Corinna
--
Antworten an o.g. (existierende) Adresse werden ungelesen verworfen.
Private Mails bitte an corinnaPLOPvinschenPINGde.

[Corinna Vinschen]

Post by r***@gmail.com

Post by Corinna Vinschen
There's never any guarantee that there's no bug in the init code, but
there's at least a 99.9% certainty that the DEP status isn't changed in
the init code.

Does Cygwin initialisation use VirtualProtect() at any point?
I had a similar sounding bug in the past which turned out to be caused
by some self-modifying code that wasn't correctly restoring the
PAGE_EXECUTE bits.

Thanks for the hint. I checked the initialization process and there are
only two cases in which VirtualProtect could be called at init time:

- Fixing inherited shared memory regions created by the parent process
using the POSIX mmap call in the child process after fork.
- Reloading shared libraries dynamically loaded by the parent process
using the LD_PRELOAD mechanism in the child process after fork.

Both cases are only triggered by a fork() and there's no fork involved
in the crashing cases. In fact, bash already crashes when called
immediately or through a cmd script. To be sure I debugged this again
and no VirtualProtect call within Cygwin gets called.

Thanks all the same. Every idea could be helpful.


Corinna

[Pavel A.]

So what is so special in calling a user function from main()?
The init code by itself calls many library subroutines,
and main itself is called ok.
Can it be something related with how the loader
interprets section names of user code vs. library routines...
are they in same .text section? Have unusual attributes?

--PA

Post by Corinna Vinschen

Post by r***@gmail.com

Post by Corinna Vinschen
There's never any guarantee that there's no bug in the init code, but
there's at least a 99.9% certainty that the DEP status isn't changed in
the init code.

Does Cygwin initialisation use VirtualProtect() at any point?
I had a similar sounding bug in the past which turned out to be caused
by some self-modifying code that wasn't correctly restoring the
PAGE_EXECUTE bits.

Thanks for the hint. I checked the initialization process and there are
- Fixing inherited shared memory regions created by the parent process
using the POSIX mmap call in the child process after fork.
- Reloading shared libraries dynamically loaded by the parent process
using the LD_PRELOAD mechanism in the child process after fork.
Both cases are only triggered by a fork() and there's no fork involved
in the crashing cases. In fact, bash already crashes when called
immediately or through a cmd script. To be sure I debugged this again
and no VirtualProtect call within Cygwin gets called.
Thanks all the same. Every idea could be helpful.
Corinna

[Corinna Vinschen]

Hi Pavel,

Post by Pavel A.
So what is so special in calling a user function from main()?

Yes, that's the basic question. Especially, why is it special only
with Terminal Services installed while it runs fine with DEP on a
non-TS server?

Post by Pavel A.
The init code by itself calls many library subroutines,
and main itself is called ok.

Right.

Post by Pavel A.
Can it be something related with how the loader
interprets section names of user code vs. library routines...
are they in same .text section? Have unusual attributes?

Not that I could see. The applications have only a small number
of segments. That's the output of `objdump -h bash.exe':

bash.exe: file format pei-i386

Sections:
Idx Name Size VMA LMA File off Algn
0 .text 000590a8 00401000 00401000 00000400 2**4
CONTENTS, ALLOC, LOAD, CODE
1 .data 00002380 0045b000 0045b000 00059600 2**5
CONTENTS, ALLOC, LOAD, DATA
2 .rdata 00012e20 0045e000 0045e000 0005ba00 2**5
CONTENTS, ALLOC, LOAD, READONLY, DATA
3 .bss 00003178 00471000 00471000 00000000 2**3
ALLOC
4 .idata 00003e3c 00475000 00475000 0006ea00 2**2
CONTENTS, ALLOC, LOAD, DATA

The start addresses of the crashing application functions are well
within the .text code segment. That's why this is so puzzeling.


Corinna

[r***@gmail.com]

Sorry the suggestion didn't help.

Just a thought -- if you run this inside windbg what does !vadump
report for the segment containing the faulting address?

Roger.

[Corinna Vinschen]

Post by r***@gmail.com
Sorry the suggestion didn't help.

No worries.

Post by r***@gmail.com
Just a thought -- if you run this inside windbg what does !vadump
report for the segment containing the faulting address?

Thanks, that was very helpful!

I just did that and the single page which contains the function is not
executable when the crash happens. This looks like a OS problem,
but read further.

For my private test application (an augmented bash built in debug mode),
the .text segment is at 0x401000 up to 46b000. `objdump -h' prints

bash-g.exe: file format pei-i386

Sections:
Idx Name Size VMA LMA File off Algn
0 .text 00069510 00401000 00401000 00000400 2**4
CONTENTS, ALLOC, LOAD, CODE
1 .data 00002380 0046b000 0046b000 00069600 2**5
CONTENTS, ALLOC, LOAD, DATA
[...]

The start address of the crashing function is 0x419d97. The printout of
!vadump for the pages in the .text segment looks like this when the crash
occurs:

BaseAddress: 00401000
RegionSize: 0000c000
State: 00001000 MEM_COMMIT
Protect: 00000080 PAGE_EXECUTE_WRITECOPY
Type: 01000000 MEM_IMAGE

BaseAddress: 0040d000
RegionSize: 00001000
State: 00001000 MEM_COMMIT
Protect: 00000040 PAGE_EXECUTE_READWRITE
Type: 01000000 MEM_IMAGE

[...]

BaseAddress: 00419000
RegionSize: 00001000
State: 00001000 MEM_COMMIT
Protect: 00000008 PAGE_WRITECOPY !!!!!
Type: 01000000 MEM_IMAGE

BaseAddress: 0041a000
RegionSize: 00002000
State: 00001000 MEM_COMMIT
Protect: 00000080 PAGE_EXECUTE_WRITECOPY
Type: 01000000 MEM_IMAGE

[...]

!vprot 0x419000 prints additionally

AllocationBase 00400000
AllocationProtect 00000080 PAGE_EXECUTE_WRITECOPY

So *something* has actually changed the protection. Not only on this
page but also on some other arbitrary pages in the .text segment.

When debugging the same on a non-TS Server 2008 machine, all pages in
the .text segment are still either PAGE_EXECUTE_WRITECOPY or
PAGE_EXECUTE_READWRITE when setting a breakpoint to the instruction
which crashes on the TS machine.

I debugged this further with WinDbg and the change of protection already
occurs before any Cygwin code ran. I set a breakpoint at the start of
the DLL entry routine and at the time it's called from the Windows
loader for the DLL_PROCESS_ATTACH rat race, the protection is already
PAGE_WRITECOPY. The test application loads three other Cygwin specific
DLLs, but as far as I can see, all these libs are loaded *after* the
Cygwin DLL. So, if I didn't miss anything, the protection of this page
has actually changed before any application code has been called.

What now? I'm not overly fluent in WinDbg since I'm using GDB all the
time. Is there a way in WinDbg to break on a page protection change?
The `ba' command can't be used before the application has been started
and I don't know if it would be triggered by a protection change anyway.
I also tried to set a breakpoint to VirtualProtect, but it's not hit
at load time.

Does that description qualify for a re-evaluation of the support case at
Microsoft Professional Support, maybe?


Corinna

[Alexander Grigoriev]

Is the function being called in a DLL or in the EXE? Does the call's target
address contain a fixup jump or actual code? Are the Cygwin DLLs loaded on
the preferred address under TS?

Post by Corinna Vinschen

Post by r***@gmail.com
Sorry the suggestion didn't help.

No worries.

Post by r***@gmail.com
Just a thought -- if you run this inside windbg what does !vadump
report for the segment containing the faulting address?

Thanks, that was very helpful!
I just did that and the single page which contains the function is not
executable when the crash happens. This looks like a OS problem,
but read further.
For my private test application (an augmented bash built in debug mode),
the .text segment is at 0x401000 up to 46b000. `objdump -h' prints
bash-g.exe: file format pei-i386
Idx Name Size VMA LMA File off Algn
0 .text 00069510 00401000 00401000 00000400 2**4
CONTENTS, ALLOC, LOAD, CODE
1 .data 00002380 0046b000 0046b000 00069600 2**5
CONTENTS, ALLOC, LOAD, DATA
[...]
The start address of the crashing function is 0x419d97. The printout of
!vadump for the pages in the .text segment looks like this when the crash
BaseAddress: 00401000
RegionSize: 0000c000
State: 00001000 MEM_COMMIT
Protect: 00000080 PAGE_EXECUTE_WRITECOPY
Type: 01000000 MEM_IMAGE
BaseAddress: 0040d000
RegionSize: 00001000
State: 00001000 MEM_COMMIT
Protect: 00000040 PAGE_EXECUTE_READWRITE
Type: 01000000 MEM_IMAGE
[...]
BaseAddress: 00419000
RegionSize: 00001000
State: 00001000 MEM_COMMIT
Protect: 00000008 PAGE_WRITECOPY !!!!!
Type: 01000000 MEM_IMAGE
BaseAddress: 0041a000
RegionSize: 00002000
State: 00001000 MEM_COMMIT
Protect: 00000080 PAGE_EXECUTE_WRITECOPY
Type: 01000000 MEM_IMAGE
[...]
!vprot 0x419000 prints additionally
AllocationBase 00400000
AllocationProtect 00000080 PAGE_EXECUTE_WRITECOPY
So *something* has actually changed the protection. Not only on this
page but also on some other arbitrary pages in the .text segment.
When debugging the same on a non-TS Server 2008 machine, all pages in
the .text segment are still either PAGE_EXECUTE_WRITECOPY or
PAGE_EXECUTE_READWRITE when setting a breakpoint to the instruction
which crashes on the TS machine.
I debugged this further with WinDbg and the change of protection already
occurs before any Cygwin code ran. I set a breakpoint at the start of
the DLL entry routine and at the time it's called from the Windows
loader for the DLL_PROCESS_ATTACH rat race, the protection is already
PAGE_WRITECOPY. The test application loads three other Cygwin specific
DLLs, but as far as I can see, all these libs are loaded *after* the
Cygwin DLL. So, if I didn't miss anything, the protection of this page
has actually changed before any application code has been called.
What now? I'm not overly fluent in WinDbg since I'm using GDB all the
time. Is there a way in WinDbg to break on a page protection change?
The `ba' command can't be used before the application has been started
and I don't know if it would be triggered by a protection change anyway.
I also tried to set a breakpoint to VirtualProtect, but it's not hit
at load time.
Does that description qualify for a re-evaluation of the support case at
Microsoft Professional Support, maybe?
Corinna
--
Corinna Vinschen
Cygwin Project Co-Leader
Red Hat

[Corinna Vinschen]

Post by Alexander Grigoriev
Is the function being called in a DLL or in the EXE? Does the call's target

In the EXE, as I described in my original posting already.

Post by Alexander Grigoriev
address contain a fixup jump or actual code? Are the Cygwin DLLs loaded on

Actual code.

Post by Alexander Grigoriev
the preferred address under TS?

Yes.


Corinna

[Corinna Vinschen]

Post by Corinna Vinschen

Post by r***@gmail.com
Sorry the suggestion didn't help.

No worries.

Post by r***@gmail.com
Just a thought -- if you run this inside windbg what does !vadump
report for the segment containing the faulting address?

Thanks, that was very helpful!
I just did that and the single page which contains the function is not
executable when the crash happens. This looks like a OS problem,
but read further.

I debugged this further with WinDbg. I set a breakpoint to
ntdll!NtProtectVirtualMemory and followed through all calls from
starting the application until the page protection of the page in
question has changed to PAGE_WRITECOPY.

It seems there's a Terminal Service component called tsappcmd.dll, which
changes the page protection of arbitrary pages in the .text segment for
some dubious reason.

Usually you see two calls to NtProtectVirtualMemory in a row, called
from tsappcmp!TermsrvUpdateAllUserMenu+0x8xx. The first one sets the
protection to READWRITE, the next one sets it back to EXECUTE_READWRITE
or EXECUTE_WRITECOPY, whatever the original state of the page was.
However, sometimes there goes something wrong, apparently. For some
pages where the original protection was EXECUTE_WRITECOPY, the first
call sets the protection to READWRITE, as usual, but the second call
sets it to just WRITECOPY instead of EXECUTE_WRITECOPY.

So the page is left in a non-executable state and a subsequent try to
execute a function located in this page crashes with a SEGV.

This all happens already at application load time. I set a breakpoint
to the Cygwin DLLs entry function. It's never hit before the protection
has already been changed by tsappcmd.dll. So all the above already
happens before any application code ran.


Corinna

[Corinna Vinschen]

Post by Corinna Vinschen

Post by Corinna Vinschen

Post by r***@gmail.com
Sorry the suggestion didn't help.

No worries.

Post by r***@gmail.com
Just a thought -- if you run this inside windbg what does !vadump
report for the segment containing the faulting address?

Thanks, that was very helpful!
I just did that and the single page which contains the function is not
executable when the crash happens. This looks like a OS problem,
but read further.

I debugged this further with WinDbg. I set a breakpoint to
ntdll!NtProtectVirtualMemory and followed through all calls from
starting the application until the page protection of the page in
question has changed to PAGE_WRITECOPY.
It seems there's a Terminal Service component called tsappcmd.dll, which

^^^^^^^^
tsappcmp, sorry.

Corinna

[Pavel A.]

How this tsappcmp.dll is loaded in your process -
does TS add it to AppInitDLLs?

The "appcmp" reminds "application compatibility", so maybe there
is some appcompat rule that matches your exe - by name
or by some exe file attributes.

--PA|

Post by Corinna Vinschen

Post by Corinna Vinschen

Post by Corinna Vinschen

Post by r***@gmail.com
Sorry the suggestion didn't help.

No worries.

Post by r***@gmail.com
Just a thought -- if you run this inside windbg what does !vadump
report for the segment containing the faulting address?

Thanks, that was very helpful!
I just did that and the single page which contains the function is not
executable when the crash happens. This looks like a OS problem,
but read further.

I debugged this further with WinDbg. I set a breakpoint to
ntdll!NtProtectVirtualMemory and followed through all calls from
starting the application until the page protection of the page in
question has changed to PAGE_WRITECOPY.
It seems there's a Terminal Service component called tsappcmd.dll, which

^^^^^^^^
tsappcmp, sorry.
Corinna

[Corinna Vinschen]

Post by Pavel A.
How this tsappcmp.dll is loaded in your process -
does TS add it to AppInitDLLs?

I don't know. I never saw this DLL before I started debugging this
problem and I never had a reason before. The AppInitDLLs value is
empty, though. I assume it's a dependency of some other DLL after
installing TS, or the Windows loader does it by itself if TS is
installed.

Post by Pavel A.
The "appcmp" reminds "application compatibility", so maybe there
is some appcompat rule that matches your exe - by name
or by some exe file attributes.

Maybe, but how am I supposed to know the rule?


Corinna

[Pavel A.]

Post by Corinna Vinschen
Maybe, but how am I supposed to know the rule?

Hmm. Seems that it doesn't help to know the rules:
they change them all the time :(

--PA

[Pavel A.]

If tsappcmp.dll indeed belongs to the application compatibility
scheme of Win2008 or TS, maybe this gives more chances with the MS support.
The appcompat stuff is designed to make 3rd party apps work, not to break
them.
So maybe you can directly contact the appcompat team.

This program on MS Connect site specially addresses TS compatibilty :

https://connect.microsoft.com/tsappcompat

( to see this page you need to log on to the Connect site )

Regards,
--PA

[Corinna Vinschen]

Post by Pavel A.
If tsappcmp.dll indeed belongs to the application compatibility
scheme of Win2008 or TS, maybe this gives more chances with the MS support.
The appcompat stuff is designed to make 3rd party apps work, not to break
them.
So maybe you can directly contact the appcompat team.
https://connect.microsoft.com/tsappcompat

Thanks for this URL! I'll try and explain the problem to them.


Corinna

[GP]

Hello Corinna!

Post by Corinna Vinschen

Post by Pavel A.
https://connect.microsoft.com/tsappcompat

Thanks for this URL! I'll try and explain the problem to them.

Maybe you can report to the group if you've successfully solved the issue
with MS.


GP

[Corinna Vinschen]

Post by GP
Hello Corinna!

Post by Corinna Vinschen

Post by Pavel A.
https://connect.microsoft.com/tsappcompat

Thanks for this URL! I'll try and explain the problem to them.

Maybe you can report to the group if you've successfully solved the issue
with MS.

Sure. Right now the MS support is still trying to find a way to avoid
support rather than trying to debug the problem. Stay tuned.

I have another open case which I originally discussed on this newsgroup
(quadratic timing behaviour accessing long paths in NT 5.x) which, after
5 months, still nobody at MS even tried to debug. I got a lecture about
correct testing instead. Well, hope never fades...


Corinna

[Alexander Grigoriev]

That's my experience with MS, too. Unless the OS crashes and burns, they
won't do anything. Even if the checked kernel ASSERTs.

Post by Corinna Vinschen

Post by GP
Hello Corinna!

Post by Corinna Vinschen

Post by Pavel A.
https://connect.microsoft.com/tsappcompat

Thanks for this URL! I'll try and explain the problem to them.

Maybe you can report to the group if you've successfully solved the issue
with MS.

Sure. Right now the MS support is still trying to find a way to avoid
support rather than trying to debug the problem. Stay tuned.
I have another open case which I originally discussed on this newsgroup
(quadratic timing behaviour accessing long paths in NT 5.x) which, after
5 months, still nobody at MS even tried to debug. I got a lecture about
correct testing instead. Well, hope never fades...
Corinna
--
Corinna Vinschen
Cygwin Project Co-Leader
Red Hat

[r***@gmail.com]

Post by Corinna Vinschen
I just did that and the single page which contains the function is not
executable when the crash happens.  This looks like a OS problem,
but read further.
For my private test application (an augmented bash built in debug mode),
the .text segment is at 0x401000 up to 46b000.  `objdump -h' prints
  bash-g.exe:     file format pei-i386
  Idx Name          Size      VMA       LMA       File off  Algn
    0 .text         00069510  00401000  00401000  00000400  2**4
                    CONTENTS, ALLOC, LOAD, CODE

Hmm, seems a bit odd that the 'READONLY' flag isn't shown.
...

Post by Corinna Vinschen
  BaseAddress:       00419000
  RegionSize:        00001000
  State:             00001000  MEM_COMMIT
  Protect:           00000008  PAGE_WRITECOPY      !!!!!
  Type:              01000000  MEM_IMAGE

Wonder whether there's an issue with the OS loader copying pages and
changing the protection?

...

Post by Corinna Vinschen
What now?  I'm not overly fluent in WinDbg since I'm using GDB all the
time.  Is there a way in WinDbg to break on a page protection change?

I don't know of one -- but what happens if you break on
NtProtectVirtualMemory in Windbg?

Post by Corinna Vinschen
Does that description qualify for a re-evaluation of the support case at
Microsoft Professional Support, maybe?

It starts to look like it...

Regards,
Roger.

[Corinna Vinschen]

Post by r***@gmail.com

Post by Corinna Vinschen
I just did that and the single page which contains the function is not
executable when the crash happens. ?This looks like a OS problem,
but read further.
For my private test application (an augmented bash built in debug mode),
the .text segment is at 0x401000 up to 46b000. ?`objdump -h' prints
? bash-g.exe: ? ? file format pei-i386
? Idx Name ? ? ? ? ?Size ? ? ?VMA ? ? ? LMA ? ? ? File off ?Algn
? ? 0 .text ? ? ? ? 00069510 ?00401000 ?00401000 ?00000400 ?2**4
? ? ? ? ? ? ? ? ? ? CONTENTS, ALLOC, LOAD, CODE

Hmm, seems a bit odd that the 'READONLY' flag isn't shown.

That seems to depend on the GNU ld version used to link the executable.
Some applications in the distro have a R/O text segment, some don't.
The applications in the distro have been built by many different people
at different times. But that doesn't explain the effect and works fine
on any other system, just not on 2008 with TS.

Post by r***@gmail.com

Post by Corinna Vinschen
? BaseAddress: ? ? ? 00419000
? RegionSize: ? ? ? ?00001000
? State: ? ? ? ? ? ? 00001000 ?MEM_COMMIT
? Protect: ? ? ? ? ? 00000008 ?PAGE_WRITECOPY ? ? ?!!!!!
? Type: ? ? ? ? ? ? ?01000000 ?MEM_IMAGE

Wonder whether there's an issue with the OS loader copying pages and
changing the protection?
...

Post by Corinna Vinschen
What now? ?I'm not overly fluent in WinDbg since I'm using GDB all the
time. ?Is there a way in WinDbg to break on a page protection change?

I don't know of one -- but what happens if you break on
NtProtectVirtualMemory in Windbg?

I did that, see my posting from about an hour ago, Message ID

Post by r***@gmail.com

Post by Corinna Vinschen
Does that description qualify for a re-evaluation of the support case at
Microsoft Professional Support, maybe?

It starts to look like it...

I have written a more detailed description of my findings to the MS
support, asking to re-open the case. Setting the page protection
apparently occurs in tsappcmp.dll, so it's not a problem in a third
party application per se, which was the reasoning to refuse the case.

I didn't get a reply yet, though...


Corinna

[Volodymyr M. Shcherbyna]

Can you provide this magic application binary(or binaries) to make some
debugging?
--
Volodymyr, blog: http://www.shcherbyna.com/
(This posting is provided "AS IS" with no warranties, and confers no
rights)

Post by Corinna Vinschen
Hi,
I'm not sure if there's another, better suited managed newsgroup for
this kind of problem. I opened a support case at Microsoft Professional
Support yesterday, but they refused to take it because the actual
problem (the crash) doesn't occur in a Windows component. The support
engineer suggested to ask in one of the managed newsgroups, so here we
are.
Months ago I tweaked Cygwin so that it works on Windows Server 2008.
There was a single problem with SEH which has been discussed in this
newsgroup back then. After this has been fixed, Cygwin worked nicely on
Server 2008. Once in while a Cygwin user claimed that Cygwin's bash
crashes on their 2008 machines, but since I was never able to reproduce
the problem, I doubted that this had anything to do with Cygwin itself.
Last week a Cygwin user noted that Cygwin's bash crashes as soon as
Terminal Services have been installed on Server 2008. I tested this
observation and could reproduce it. Bash, as well as some other
applications (GDB, grep) crash, but most other applications still run
The crash occurs in plain application code, not in the Cygwin DLL. The
application's main() function calls some arbitrary other application
function and the first assembler instruction in this called function
crashes with a STATUS_ACCESS_VIOLATION. Analyzing this exception with
WinDbg shows that the reason for the SEGV is an "Attempt to execute
non-executable address XYZ", with XYZ the address of the first (and
crashing) assembler instruction in the called function.
The address in question is well within the .text segment of the
executable. There's no visible reason why this address should be a
non-executable address.
As it turned out, this weird crash only occurs when the application is
running with DEP switched on (default on Server 2008) *and* Terminal
Services are installed. If you switch off DEP, or you deinstall
Terminal Services, the crash doesn't occur. It doesn't matter whether
Terminal Services are running or not, just the fact that they are
installed is sufficient to trigger the crash. You can deinstall and
reinstall TS multiple times and the behaviour is reproducible, no crash
without TS, crash with TS. It also does not matter whether you install
Cygwin before TS is installed or afterwards. Cygwin has no dependencies
to special TS properties anyway.
For testing purposes I switched off parts of the Cygwin initialization
code one at a time, for instance installing SEH handler, Ctrl-C handler,
stuff like that, but to no avail.
So, now I'm at a loss what to look for. I have a hacky workaround which
is to switch off DEP in the Cygwin initialization code, but that's
really just a hack, IMO. Of course that only works if DEP is running in
OptOut mode.
Any help, *especially* (but not only) from Microsoft, would be really
appreciated.
Corinna
--
Corinna Vinschen
Cygwin Project Co-Leader
Red Hat

[Corinna Vinschen]

Post by Volodymyr M. Shcherbyna
Can you provide this magic application binary(or binaries) to make some
debugging?

Sure! You can simply download a base Cygwin installation using the
setup tool provided at http://cygwin.com/ Bash is always part of a base
install and started when double clicking on the Cygwin icon installed on
your desktop.

If you need help, feel free to contact me off-group (vinschen at redhat
dot com).


Corinna

[Alexander Grigoriev]

Check if SeCreateGlobalPrivilege was disabled after Terminal Services
installation.

Post by Corinna Vinschen
Hi,
I'm not sure if there's another, better suited managed newsgroup for
this kind of problem. I opened a support case at Microsoft Professional
Support yesterday, but they refused to take it because the actual
problem (the crash) doesn't occur in a Windows component. The support
engineer suggested to ask in one of the managed newsgroups, so here we
are.
Months ago I tweaked Cygwin so that it works on Windows Server 2008.
There was a single problem with SEH which has been discussed in this
newsgroup back then. After this has been fixed, Cygwin worked nicely on
Server 2008. Once in while a Cygwin user claimed that Cygwin's bash
crashes on their 2008 machines, but since I was never able to reproduce
the problem, I doubted that this had anything to do with Cygwin itself.
Last week a Cygwin user noted that Cygwin's bash crashes as soon as
Terminal Services have been installed on Server 2008. I tested this
observation and could reproduce it. Bash, as well as some other
applications (GDB, grep) crash, but most other applications still run
The crash occurs in plain application code, not in the Cygwin DLL. The
application's main() function calls some arbitrary other application
function and the first assembler instruction in this called function
crashes with a STATUS_ACCESS_VIOLATION. Analyzing this exception with
WinDbg shows that the reason for the SEGV is an "Attempt to execute
non-executable address XYZ", with XYZ the address of the first (and
crashing) assembler instruction in the called function.
The address in question is well within the .text segment of the
executable. There's no visible reason why this address should be a
non-executable address.
As it turned out, this weird crash only occurs when the application is
running with DEP switched on (default on Server 2008) *and* Terminal
Services are installed. If you switch off DEP, or you deinstall
Terminal Services, the crash doesn't occur. It doesn't matter whether
Terminal Services are running or not, just the fact that they are
installed is sufficient to trigger the crash. You can deinstall and
reinstall TS multiple times and the behaviour is reproducible, no crash
without TS, crash with TS. It also does not matter whether you install
Cygwin before TS is installed or afterwards. Cygwin has no dependencies
to special TS properties anyway.
For testing purposes I switched off parts of the Cygwin initialization
code one at a time, for instance installing SEH handler, Ctrl-C handler,
stuff like that, but to no avail.
So, now I'm at a loss what to look for. I have a hacky workaround which
is to switch off DEP in the Cygwin initialization code, but that's
really just a hack, IMO. Of course that only works if DEP is running in
OptOut mode.
Any help, *especially* (but not only) from Microsoft, would be really
appreciated.
Corinna
--
Corinna Vinschen
Cygwin Project Co-Leader
Red Hat

[Corinna Vinschen]

Post by Alexander Grigoriev
Check if SeCreateGlobalPrivilege was disabled after Terminal Services
installation.

SeCreateGlobalPrivilege is not required for a Cygwin process to run.
In fact this is even tested in the DLL and taken into account when
creating shared memory. Well, in the current release, the next major
release will create shared mem completely different.

I don't know how others experienced this problem. Personally I was
testing with an admin account with UAC running. The shell was started
in restricted mode, so the SeCreateGlobalPrivilege wasn't in the user
token anyway. Other than that, the default SeCreateGlobalPrivilege
settings in the Security Policy are the same on my TS machine as on
another non-TS machine and also didn't change when deinstalling/
reinstalling TS on the test machine.

I'm a bit puzzeled because I don't see the connection between this sort
of crash and the SeCreateGlobalPrivilege. Would you mind to explain?


Thanks,
Corinna