How determine application subsystem from executable file

Question

I'm trying to detect console application from the list of the executables files installed on my computer.

How to implement it?

Every application has a "subsystem" (windows application, console application or library; specified to the linker as option, I think). How to detect it using only the executable file?

Are there alternative methods to detect the application characteristic? Additionally, are there any method for detecting the file is a really executable file?

Any issue for JAR executables?

Answer 1

Without any programming you receive this information from

dumpbin.exe /headers filename

Some information gives you GetBinaryType and SHGetFileInfo functions. All information which you needs you will find in the header of every executable file. See Microsoft Portable Executable and Common Object File Format Specification in http://www.microsoft.com/whdc/system/platform/firmware/PECOFF.mspx.

One can use also Debug Help Library API from DbgHelp.dll (see http://msdn.microsoft.com/en-us/library/ms679309(VS.85).aspx). IMAGE_DOS_HEADER, IMAGE_DOS_SIGNATURE and IMAGE_NT_HEADERS32 structures gives you full information.

UPDATED (add some code): Or you can use only structures defined in WinNT.h. The corresponding code can start like following

// Open source file
hSrcFile = CreateFile (pszSrcFilename, GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, 0, NULL);
if (hSrcFile == INVALID_HANDLE_VALUE)
    __leave;

// Map the source file in memory
hMapSrcFile = CreateFileMapping (hSrcFile, NULL, PAGE_READONLY, 0, 0, NULL);    // SEC_IMAGE
if (!hMapSrcFile || hMapSrcFile == INVALID_HANDLE_VALUE)
    __leave;

// Map the entire of the source file is memory
pSrcFile = (PBYTE) MapViewOfFile (hMapSrcFile, FILE_MAP_READ, 0, 0, 0);
if (!pSrcFile)
    __leave;

pDosHeader = (IMAGE_DOS_HEADER *)pSrcFile;

if (pDosHeader->e_magic != IMAGE_DOS_SIGNATURE) {
    printf ("it is not a EXE file.\n");
    return 1;
}
printf ("IMAGE_DOS_HEADER size %d (0x%X) bytes\n",  sizeof(IMAGE_DOS_HEADER), sizeof(IMAGE_DOS_HEADER));

DumpDosHeader (pDosHeader);
pDosExeStart = (PBYTE)pDosHeader + pDosHeader->e_cparhdr*16;

if (g_bDump)
    HexDump (1, pDosExeStart, pDosHeader->e_lfanew - pDosHeader->e_cparhdr*16, (DWORD)pDosExeStart);

if (pDosHeader->e_lfanew) {
    IMAGE_NT_HEADERS32 *pNtHeader = (IMAGE_NT_HEADERS32 *)((PBYTE)pDosHeader + pDosHeader->e_lfanew);
    //IMAGE_NT_HEADERS64 *pNtHeader64 = (IMAGE_NT_HEADERS64 *)((PBYTE)pDosHeader + pDosHeader->e_lfanew);
    IMAGE_SECTION_HEADER *pFirstSectionHeader = (IMAGE_SECTION_HEADER *)((PBYTE)&pNtHeader->OptionalHeader +
                                                                            pNtHeader->FileHeader.SizeOfOptionalHeader);

    if (pNtHeader->Signature == IMAGE_NT_SIGNATURE) {
        int i;

        printf ("\nPE signature\n");
        printf ("\nIMAGE_FILE_HEADER: size %d (0x%X) bytes, offset from the begin of the file: %d (0x%X)\n",
                sizeof(IMAGE_FILE_HEADER), sizeof(IMAGE_FILE_HEADER),
                ((PBYTE)&pNtHeader->FileHeader - (PBYTE)pDosHeader), ((PBYTE)&pNtHeader->FileHeader - (PBYTE)pDosHeader));
        DumpFileHeader (1, &pNtHeader->FileHeader);
        switch (pNtHeader->OptionalHeader.Magic) {
            case IMAGE_NT_OPTIONAL_HDR32_MAGIC:
                printf ("\nIMAGE_OPTIONAL_HEADER32: size %d (0x%X) bytes, offset from the begin of the file: %d (0x%X)\n",
                        sizeof(IMAGE_OPTIONAL_HEADER32), sizeof(IMAGE_OPTIONAL_HEADER32),
                        ((PBYTE)&pNtHeader->OptionalHeader - (PBYTE)pDosHeader), ((PBYTE)&pNtHeader->OptionalHeader - (PBYTE)pDosHeader));
                DumpOptionalHeader32 (1, &pNtHeader->OptionalHeader);
                break;
            case IMAGE_NT_OPTIONAL_HDR64_MAGIC:
                break;
            case IMAGE_ROM_OPTIONAL_HDR_MAGIC:
                break;
        }

Answer 2

Windows PE executables have a field in the header that specify the subsystem (Console, GUI, Posix, and so on). They also have fields that can be used to identify executables in general. Download the PE specification from msdn.com to get the details.

Answer 3

To determine the subsystem you need to read the executable file and parse the PE-Header. A detailed article on how to do that is found here.

A JAR file is just a ZIP file with some specific files and folder structure so you can open it like a regular zip file and look for those files and folders that are always there.