Visual C++ 6.0 on Windows 8

Question

Visual C++ 6.0 is not supported on Windows 8, but we have a couple of legacy apps that still needs to be compiled with Visual C++ 6.0. :-(

It is possible to install Visual C++ 6.0 on Windows 8 by unchecking Data Access -> Change Options -> ADO, RDS and OLE DB Providers. See this SU-question and this thread. You also need to install SP6 afterwards.

Visual C++ 6.0 works perfectly on one computer, but two others cannot use the debugger. The same hardware, same version of Windows, same person doing the installation, same project. There must be some difference...

On the computers with the problem you can set a break point and the debugger will break into the IDE, but when you try do step, step into or run the code will crash with Unhandled exception in EXENAME.EXE (OLE32.DLL): 0xC0000005: Access Violation.

Walter Oney reports the exact same problem on MSDN forums, but they have no solution as VC++ 6.0 is unsupported.

As we have Visual C++ 6.0 working on one Win8 computer there is apparently way to do it. Any ideas on what the difference could be?

Answer 1

Turning off OLE RPC debugging (Tools / Options / Debug) works for me (Windows 8 Pro 64 bit, Visual C++ 6.0 with SP6). This solution was suggested (later) within the above-mentioned MSDN forum thread.

Answer 2

I was eventually able to get VS 6 working on Win 8 and Win 10. The basic steps were these:

1. Create a dummy file named msjava.dll in \Windows. (E.g., "echo >msjava.dll") Without this step, the VS 6 installer can't get very far.

2. Install VS 6 and SP 6.

3. Rename MSDEV.EXE to something else, such as MSDEVQ.EXE.

4. Create a compatibility database for MSDEVQ that excludes the fault-tolerant heap shim. Without this step, debugging a program that makes heavy use of HeapAlloc, etc., is excruciatingly slow.

5. For debugging, ensure that a breakpoint is tripped before any calls to OLE32 can occur. I include the following header early in the main program or (for an MFC app) the InitInstance function:

X64DebugHack.h:

#ifdef _DEBUG
// In order to be able to debug this application on x64, we need to single
// step across at least one statement before ole32.dll gets loaded. So
// always leave this breakpoint in place.

requiredbreakpoint:
    int junkola = 42;

    // Check to see that there was a breakpoint...

    PUCHAR pjunk;
    _asm lea eax, requiredbreakpoint
    _asm mov pjunk, eax

    if (*pjunk != 0xCC)
        AfxMessageBox("Required breakpoint was not set prior to loading OLE32.DLL -- single stepping will not be possible during this debugging session.", MB_OK | MB_ICONHAND, 0);

    LoadLibrary("OLE32");
#endif

6. Write an extension DLL that provides a "Stop Debugging" button. The extension has to search and destroy debug handles, which have a different handle type in Win64 than in Win32. The mechanics of writing the extension are beyond the scope of this forum, but the code that does the actual work is here:

CCommands::HelpAssistantKill:

typedef LONG NTSTATUS;
#define NT_SUCCESS(Status)  (((NTSTATUS)(Status)) >= 0)
#define STATUS_INFO_LENGTH_MISMATCH      ((NTSTATUS)0xC0000004L)

enum SYSTEM_INFORMATION_CLASS {
    SystemHandleInformation = 16,
    };

typedef NTSTATUS(NTAPI *PNTQUERYSYSTEMINFORMATION)(SYSTEM_INFORMATION_CLASS, PVOID, ULONG, PULONG);

typedef struct _SYSTEM_HANDLE_INFORMATION {
    ULONG ProcessId;
    UCHAR ObjectTypeNumber;
    UCHAR Flags;
    USHORT Handle;
    PVOID Object;
    ACCESS_MASK GrantedAccess;
    } SYSTEM_HANDLE_INFORMATION, *PSYSTEM_HANDLE_INFORMATION;

typedef struct _SYSTEM_HANDLE_INFORMATION_DATA {
    ULONG HandleCount;
    SYSTEM_HANDLE_INFORMATION HandleInformation[1];
    } SYSTEM_HANDLE_INFORMATION_DATA, *PSYSTEM_HANDLE_INFORMATION_DATA;

#define HANDLE_TYPE_DEBUG_OBJECT 11     // correct value for Win8 x64

STDMETHODIMP CCommands::HelpAssistantKill()
    {                           // CCommands::HelpAssistantKill
    AFX_MANAGE_STATE(AfxGetStaticModuleState());

    BOOL didit = FALSE;
    HMODULE hDll = NULL;
    PSYSTEM_HANDLE_INFORMATION_DATA phi = NULL;

    do  {                       // do once
        HRESULT hr;

        // Locate NtQuerySystemInformation within NTDLL.DLL

        hDll = LoadLibrary("NTDLL");
        if (!hDll)
            break;

        PNTQUERYSYSTEMINFORMATION NtQuerySystemInformation = (PNTQUERYSYSTEMINFORMATION) GetProcAddress(hDll, "NtQuerySystemInformation");
        if (!NtQuerySystemInformation)
            break;

        // Do an initial query to get the number of handles presently open in the system.
        // This is a large number. The returned length value is meaningless for this query.

        SYSTEM_HANDLE_INFORMATION_DATA hid;
        DWORD junk;
        NTSTATUS status = (*NtQuerySystemInformation)(SystemHandleInformation, &hid, sizeof(hid), &junk);
        if (!NT_SUCCESS(status) && status != STATUS_INFO_LENGTH_MISMATCH)
            break;

        ULONG length = sizeof(SYSTEM_HANDLE_INFORMATION_DATA) + (hid.HandleCount - 1) * sizeof(SYSTEM_HANDLE_INFORMATION);
        phi = (PSYSTEM_HANDLE_INFORMATION_DATA) new UCHAR[length];
        if (!phi)
            break;

        // Get a list of all handles open in the system

        status = (*NtQuerySystemInformation)(SystemHandleInformation, phi, length, &junk);
        if (!NT_SUCCESS(status))
            break;

        // Find and close any debug objects that are open in this instance of Visual Studio.

        DWORD pid = GetCurrentProcessId();
        ULONG ihandle;
        for (ihandle = 0; ihandle < hid.HandleCount; ++ihandle)
            {                   // for each open handle
            PSYSTEM_HANDLE_INFORMATION p = phi->HandleInformation + ihandle;
            if (p->ProcessId != pid || p->ObjectTypeNumber != HANDLE_TYPE_DEBUG_OBJECT)
                continue;

            if (CloseHandle((HANDLE) p->Handle))
                didit = TRUE;           
            }                   // for each open handle

        // Instruct DevStudio to stop

        BSTR bsStopDebugging = SysAllocString(L"DebugStopDebugging");
        if (!bsStopDebugging)
            break;

        hr = m_pApplication->ExecuteCommand(bsStopDebugging);
        SysFreeString(bsStopDebugging);
        if (hr != 0)
            break;
        }                       // do once
    while (FALSE);

    if (phi)
        delete[] phi;

    if (hDll)
        FreeLibrary(hDll);

    if (!didit)
        {                       // didn't do anything
        MessageBox(NULL, "Unable to find and close any debug object handles", "HelpAssistant", MB_OK | MB_ICONINFORMATION);
        }                       // didn't do anything

    return S_OK;
    }                           // CCommands::HelpAssistantKill

This felt like a pretty heroic effort, but I had about a million lines of code built on VS 6 that I had to keep working. Now that I've built myself a workable macro processor for VS 2015, I may undertake a conversion of this application.