AttachConsole and 64 bit application

Question

The WinAPI function AttachConsole always returns true when the program is compiled as 64-bit.

First of all, I've declared the function as following:

function AttachConsole(dwProcessId: DWORD): Bool; stdcall; external KERNEL32 name 'AttachConsole';

Then I call my function:

if AttachConsole(DWORD(-1)) then
   ....

This works fine when compiled as a 32-bit application, but when compiled as 64-bit it always returns true.

The documentation doesn't mention doing something special for a 64-bit application.

How to reproduce:

1. Create a new VCL Application
2. Set Target platform to Win64
3. Edit the DPR file to look like this:

program Project1;

uses
  System.Types,
  WinApi.windows,
  Vcl.Forms,
  Unit1 in 'Unit1.pas' {Form1};

{$R *.res}

function AttachConsole(dwProcessId: DWORD): Bool; stdcall; external KERNEL32 name 'AttachConsole';


begin
  if AttachConsole(DWORD(-1)) then
  begin
    writeLN('Hello world');
    Exit;
  end;

  Application.Initialize;
  Application.MainFormOnTaskbar := True;
  Application.CreateForm(TForm1, Form1);
  Application.Run;
end.

When running under Win64, AttachConsole retruns true even when run from Explorer.

Answer 1

From the documenation link, dwProcessId [in] can take two types of values - either a PID of the target process or the special argument :

ATTACH_PARENT_PROCESS (DWORD)-1

Use the console of the parent of the current process.

Here you are using the value ATTACH_PARENT_PROCESS.

In the case of a 64-bit debug, it appears that the IDE is creating a console for the 64-bit debugger, which hosts the debugged application as a child, and so electing to attach to the parent console succeeds.

For a 32-bit debug session the application is spawned as a child of the IDE. We can guess that this is because the IDE itself is a 32-bit application and can hook into the 32-bit process directly while the 64-bit application must be attached to a 64-bit debugger.

Using process explorer we can see the difference in the process hierarchy when launching the process for debugging within the IDE :

Here we can see the 64-bit process is hosted as a child in the debugger kernel while the 32-bit process is not.

Running the (64-bit) application outside of the debugger produces the expected result where AttachConsole(ATTACH_PARENT_PROCESS) fails.