CreateFile with the FILE_FLAG_DELETE_ON_CLOSE flag

Question

Before I describe my problem, here is a description of the program (IHExplorer.exe) I'm writting:

This is a C++ application.

The IHExplorer application is to look as much like a Windows Explorer window as possible. With one exception, and that is that launching files from within this Explorer window will decrypt them first to the user's temp folder, then launch the app associated with the file extension and delete the file on close.

The problem i'm having is with the auto delete when the file is closed. Here's a scenario:

1. User double clicks an encrypted .txt file in IHExplorer.

2. IHExplorer decrypts the .txt file in memory, then writes it to %TEMP% using ::CreateFile which returns a HANDLE to the file (IHExplorer has to keep this handle open atleast until the .txt file is shell executed).

3. IHExplorer Shell Executes the .txt file (by calling ::ShellExecute) from it's temp location.

4. Now IHExplorer and notepad both have a handle to the file open.
5. The file must be auto deleted when both IHExplorer and notepad have both closed their handle to the file, even if IHExplorer closes first.

ok. that is a basical User Case that describes what I want to happen. The problem I have is when I ::ShellExecute(), notepad says "The process cannot access the file because it is being used by another process." (which would be IHExplorer). I need to get around this and have notepad open it even while I still have the handle open in IHExplorer.

Here's what my call to ::CreateFile looks like:

DWORD dwShareMode = FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE;
DWORD dwFlagsAndAttributes = FILE_ATTRIBUTE_TEMPORARY | FILE_FLAG_DELETE_ON_CLOSE;
HANDLE hFile = ::CreateFile(strTempFile.c_str(), GENERIC_WRITE, dwShareMode, &sa, CREATE_NEW, dwFlagsAndAttributes, NULL);

Notice I used FILE_SHARE_DELETE so that other processes (such as notepad) can open the file with delete access.

Notice that I used the FILE_ATTRIBUTE_TEMPORARY | FILE_FLAG_DELETE_ON_CLOSE attributes to indicicate the the file is temporary and should be deleted on close.

Also notice the &sa parameter. This is the SECURITY_ATTRIBUTES structure that I am using, and I feel (hope) this is where my problem lies. Here is the code again, this time I will post the entire function so you can see how I fill out the SECURITY_ATTRIBUTES structure:

int CIHExplorerDoc::OpenFile(std::string strFileName, bool bIsFullPath) {
    std::string strFullFilePath;
    if(bIsFullPath) {
        strFullFilePath = strFileName;
        strFileName = IHawk::RemovePath(strFileName);
    }else {
        strFullFilePath = m_strDirectory + strFileName;
    }

    if(!HasEncryptionFileExtension(strFullFilePath)) {
        LaunchFile(strFullFilePath);
    }else {
        //it's an encrypted file, so open it and copy unencrypted file to temp.
        IHawk::EncryptedFileHandle hEncryptedFile(strFullFilePath.c_str(), true, theApp.GetKeyServer());
        if(hEncryptedFile.IsValid()) {
            std::string strTempFile = g_strTempFolder + IHawk::ChangeFileExtension(strFileName, "");

            //TODO: Determine what the LPSECURITY_ATTRIBUTES should be.

            SECURITY_ATTRIBUTES sa;
            SECURITY_DESCRIPTOR sd;

            if(!InitializeSecurityDescriptor(&sd,SECURITY_DESCRIPTOR_REVISION)) {
                DWORD dwLastError = ::GetLastError();
                LOG4CPLUS_ERROR(m_Logger, "Cannot launch file '" << strFullFilePath << "'.  Failed to initialize security descriptor.  GetLastError=" << dwLastError);
                return dwLastError;
            }

            if(!SetSecurityDescriptorDacl(
                &sd,    // A pointer to the SECURITY_DESCRIPTOR structure to which the function adds the DACL
                TRUE,   // presence of a DACL in the security descriptor
                NULL,   // allows all access to the object
                FALSE   // DACL has been explicitly specified by a user
            )) 
            {
                DWORD dwLastError = ::GetLastError();
                LOG4CPLUS_ERROR(m_Logger, "Cannot launch file '" << strFullFilePath << "'.  Failed to set security descriptor DACL.  GetLastError=" << dwLastError);
                return dwLastError;
            }

            if(!SetSecurityDescriptorGroup(
                &sd,    // A pointer to the SECURITY_DESCRIPTOR structure whose primary group is set by this function
                NULL,   // no primary group
                FALSE   // Indicates whether the primary group information was derived from a default mechanism
            ))
            {
                DWORD dwLastError = ::GetLastError();
                LOG4CPLUS_ERROR(m_Logger, "Cannot launch file '" << strFullFilePath << "'.  Failed to set security descriptor primary group.  GetLastError=" << dwLastError);
                return dwLastError;
            }

            if(!SetSecurityDescriptorOwner(
                &sd,    // A pointer to the SECURITY_DESCRIPTOR structure whose owner is set by this function.
                NULL,   // If this parameter is NULL, the function clears the security descriptor's owner information. This marks the security descriptor as having no owner.
                FALSE   // Indicates whether the owner information is derived from a default mechanism.
            ))
            {
                DWORD dwLastError = ::GetLastError();
                LOG4CPLUS_ERROR(m_Logger, "Cannot launch file '" << strFullFilePath << "'.  Failed to set security descriptor owner information.  GetLastError=" << dwLastError);
                return dwLastError;
            }

            if(!SetSecurityDescriptorSacl(
                &sd,    // A pointer to the SECURITY_DESCRIPTOR structure to which the function adds the SACL
                FALSE,  // the security descriptor does not contain a SACL
                NULL,   // security descriptor has a NULL SACL
                FALSE   // A pointer to a flag that is set to the value of the SE_SACL_DEFAULTED flag in the SECURITY_DESCRIPTOR_CONTROL structure if a SACL exists for the security descriptor
            ))
            {
                DWORD dwLastError = ::GetLastError();
                LOG4CPLUS_ERROR(m_Logger, "Cannot launch file '" << strFullFilePath << "'.  Failed to set security descriptor SACL.  GetLastError=" << dwLastError);
                return dwLastError;
            }

            sa.nLength = sizeof(SECURITY_ATTRIBUTES);
            sa.lpSecurityDescriptor = &sd;
            sa.bInheritHandle = TRUE;

            DWORD dwShareMode = FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE;
//          DWORD dwFlagsAndAttributes = FILE_ATTRIBUTE_NORMAL;
            DWORD dwFlagsAndAttributes = FILE_ATTRIBUTE_TEMPORARY | FILE_FLAG_DELETE_ON_CLOSE;
            HANDLE hFile = ::CreateFile(strTempFile.c_str(), GENERIC_WRITE, dwShareMode, &sa, CREATE_NEW, dwFlagsAndAttributes, NULL);

            //verify we created the file.
            if(hFile == INVALID_HANDLE_VALUE) {
                DWORD dwLastError = ::GetLastError();
                return dwLastError;
            }

            //copy to temp
            char buffer[64*1024];
            size_t nBytesRead = hEncryptedFile.Read(buffer, sizeof(buffer));
            while(nBytesRead) {
                DWORD numBytesWritten;
                if(!::WriteFile(hFile, buffer, nBytesRead, &numBytesWritten, (LPOVERLAPPED) NULL)) {
                    DWORD dwLastError = ::GetLastError();
                    LOG4CPLUS_ERROR(m_Logger, "Failed to write file to %TEMP% folder.  GetLastError=" << dwLastError);
                    return dwLastError;
                }
                nBytesRead = hEncryptedFile.Read(buffer, sizeof(buffer));
            }
            hEncryptedFile.Close();

            //execute the file from temp.
            LaunchFile(strTempFile);
        }
    }
    return 0;
}

I think if I determine the correct SECURITY_DESCRIPTOR to pass to ::CreateFile it may work like I want it to. Please help.

btw, the LaunchFile function just ends up calling ::ShellExecute to launch the file.

Answer 1

After re-reading the msdn doc, I fear I have answered my own question. FILE_FLAG_DELETE_ON_CLOSE The file is to be deleted immediately after all of its handles are closed, which includes the specified handle and any other open or duplicated handles. If there are existing open handles to a file, the call fails unless they were all opened with the FILE_SHARE_DELETE share mode. Subsequent open requests for the file fail, unless the FILE_SHARE_DELETE share mode is specified. In my case, I doubt notepad is requesting the FILE_SHARE_DELETE permission, so it can't open the file