Delphi TOpenDialog hangs in windows 2008 when run as remote desktop application

Question

I have a Delphi 2010 exe that launches a second exe. In the second exe, there is a dialog that calls openDialog.execute. When this runs under Windows 2008 Enterprise R2 under a remote desktop, it runs as expected, but when run as a remote application, as soon as the file dialog pops up, the application hangs, turning all of the application windows white. The only way to get out of it is to terminate the application. I tried replacing TOpenDialog with TFileOpenDialog, the results are the same. I've looked into modifying the RDP file that launches the main application, but cannot see any parameters there that would make a difference. Has anyone ever seen this kind of behavior before?

---

2010.07.13 Updated

This is reproducable using a simple example. There are two executable files in the example. The first is a file launcher, called m_module.exe, which contains one edit, one button, and the code below. I change the name of the executable file in the edit to match the second executable before I click the launch button:

procedure TForm1.Button1Click(Sender: TObject);
begin
     ShellExecute(Handle, 'open', stringToOLEstr(edit1.text) , nil, nil, SW_SHOWNORMAL) ; 
end;

procedure TForm1.FormShow(Sender: TObject);
begin
     edit1.text:=application.exename;
end;

The second executable contains a button and the code below:

procedure TForm1.Button1Click(Sender: TObject);
begin
     OpenDialog1.execute;
end;

The first module is launched from an RDP file.

2010.07.14 Updated

I have discovered that if I copy the following dlls:

thumbcache.dll 
dtsh.dll 
wkscli.dll 

from the \Windows\System32 folder into the application folder, the problem is eliminated.

I've further discovered that changing ownership and permission levels of these dlls in the \Windows\System32 folder from TrustedInstaller to the Administrator's group has the same result (Copying them to the application directory is changing ownership and permission I think)

To confirm this, I verified that the errors reappeared if I changed the ownership and permission levels back to TrustedInstaller away from the Administrator's group.

So it appears that this is an access issue of some kind. Perhaps this will help in discovering the cause of the issue.

2010.07.18 Updated

Some additional information that might be helpful (provided by Embarcadero):

This MSDN article for GetWindowsDirectory http://msdn.microsoft.com/en-us/library/ms724454%28VS.85%29.aspx documents some interesting behavior of applications running under Terminal Services. While GetWindowsDirectory is not being called directly the sandboxing of the Windows System directory per user could be causing some sort of problem. Perhaps one of the DLLs in the calling chain to GetOpenFileNameA is trying to reference the real DLL in the real System directory instead of the sandboxed one thus causing a rights violation. It is just speculation but it is worth investigating. If you were able to get the SysInternals Process Monitor or Process Explorer working on the server you should be able to see commdlg32 and the other DLLs in the stack trace being loaded.

All legacy applications (i.e. all applications not created for Terminal Services or Remote Desktop Services) run under an Application Compatibility Layer. See this MSDN article http://msdn.microsoft.com/en-us/library/cc834995%28VS.85%29.aspx . The IMAGE_DLLCHARACTERISTICS_TERMINAL_SERVER_AWARE flag is defined in Windows.PAS. For testing purposes you can add it to your application's PE header by adding Windows to your application's USES section and right under the USES section put:

{$SetPEOptFlags IMAGE_DLLCHARACTERISTICS_TERMINAL_SERVER_AWARE}

This will cause your application to bypass the compatibility layer. I am currently investigating if spawned processes (e.g. your second exe) retain all of the rights and settings of the application defined under RDS.

Answer 1

Windows reports AV (c0000005) in thumbcache.dll module.

I think that thumbcache.dll have something to do with building/caching thumbnails for files. Building thumbnails may mean using 3rd party extensions to Explorer, which may not behave nicely with RDP.

Try that on clean system. Use VMWare or similar virtual machine to setup test configuration.

P.S. See also this article: How to debug application’s hang? But I think that hang is just consequence of another problem in your case.