Is there a library available which easily can record and replay results of API calls?

Question

I'm working on writing various things that call relatively complicated Win32 API functions. Here's an example:

//Encapsulates calling NtQuerySystemInformation buffer management.
WindowsApi::AutoArray NtDll::NtQuerySystemInformation( 
    SystemInformationClass toGet ) const
{
    AutoArray result;
    ULONG allocationSize = 1024;
    ULONG previousSize;
    NTSTATUS errorCheck;
    do 
    {
        previousSize = allocationSize;
        result.Allocate(allocationSize);
        errorCheck = WinQuerySystemInformation(toGet, 
            result.GetAs<void>(), allocationSize, &allocationSize);
        if (allocationSize <= previousSize)
            allocationSize = previousSize * 2;
    } while (errorCheck == 0xC0000004L);
    if (errorCheck != 0)
    {
        THROW_MANUAL_WINDOWS_ERROR(WinRtlNtStatusToDosError(errorCheck));
    }
    return result;
}
//Client of the above.
ProcessSnapshot::ProcessSnapshot()
{
    using Dll::NtDll;
    NtDll ntdll;
    AutoArray systemInfoBuffer = ntdll.NtQuerySystemInformation(
        NtDll::SystemProcessInformation);
    BYTE * currentPtr = systemInfoBuffer.GetAs<BYTE>();
    //Loop through the results, creating Process objects.
    SYSTEM_PROCESSES * asSysInfo;
    do
    {
        // Loop book keeping
        asSysInfo = reinterpret_cast<SYSTEM_PROCESSES *>(currentPtr);
        currentPtr += asSysInfo->NextEntryDelta;

        //Create the process for the current iteration and fill it with data.
        std::auto_ptr<ProcImpl> currentProc(ProcFactory(
            static_cast<unsigned __int32>(asSysInfo->ProcessId), this));
        NormalProcess* nptr = dynamic_cast<NormalProcess*>(currentProc.get());
        if (nptr)
        {
            nptr->SetProcessName(asSysInfo->ProcessName);
        }
        // Populate process threads
        for(ULONG idx = 0; idx < asSysInfo->ThreadCount; ++idx)
        {
            SYSTEM_THREADS& sysThread = asSysInfo->Threads[idx];
            Thread thread(
                currentProc.get(),
                static_cast<unsigned __int32>(sysThread.ClientId.UniqueThread),
                sysThread.StartAddress);
            currentProc->AddThread(thread);
        }
        processes.push_back(currentProc);
    } while(asSysInfo->NextEntryDelta != 0);
}

My problem is in mocking out the NtDll::NtQuerySystemInformation method -- namely, that the data structure returned is complicated (Well, here it's actually relatively simple but it can be complicated), and writing a test which builds the data structure like the API call does can take 5-6 times as long as writing the code that uses the API.

What I'd like to do is take a call to the API, and record it somehow, so that I can return that recorded value to the code under test without actually calling the API. The returned structures cannot simply be memcpy'd, because they often contain inner pointers (pointers to other locations in the same buffer). The library in question would need to check for these kinds of things, and be able to restore pointer values to a similar buffer upon replay. (i.e. check each pointer sized value if it could be interpreted as a pointer within the buffer, change that to an offset, and remember to change it back to a pointer on replay -- a false positive rate here is acceptable)

Is there anything out there that does anything like this?

Answer 1

Is it really that hard to implement by hand?

#include <stddef.h>
#include <assert.h>
#include <stdlib.h>
#include <stdio.h>

typedef char BYTE;

//Note: Buffer is MODIFIED after this call (hence why it's not const)! Don't re-use it!
void saveBuffer(FILE* stream, BYTE* buffer, size_t bufferSize)
{
    assert(bufferSize % sizeof(void*) == 0);
    fwrite(&bufferSize, sizeof(bufferSize), 1, stream);
    for (size_t i = 0; i < bufferSize - sizeof(void*) + 1; i += sizeof(void*))
    {
        void** pAsPointer = (void**)&buffer[i];
        if (*pAsPointer >= buffer && *pAsPointer < buffer + bufferSize)
        {
            fwrite(&i, sizeof(i), 1, stream);
            *pAsPointer = pAsPointer;
        }
    }
    void* null = NULL;
    fwrite(&null, sizeof(null), 1, stream); //Null-terminator
    fwrite(buffer, 1, bufferSize, stream);
}


size_t loadBuffer(FILE* stream, BYTE* buffer, size_t bufferCapacity) //Buffer size is in stream
{
    size_t bufferSize;
    fread(&bufferSize, sizeof(bufferSize), 1, stream);
    if (bufferSize >= bufferCapacity)
    {
        memset(buffer, 0, bufferSize);
        for (;;)
        {
            size_t pointerOffset;
            fread(&pointerOffset, sizeof(pointerOffset), 1, stream);
            if (pointerOffset != 0)
            { *(size_t*)&buffer[pointerOffset] = (size_t)buffer + pointerOffset; }
            else { break; }
        }
        for (size_t i = 0; i < bufferSize; i += sizeof(void*))
        {
            if (*(void**)&buffer[i] == NULL)
            { fread(&buffer[i], sizeof(size_t), 1, stream); }
        }
    }
    return bufferSize;
}

(Sorry, I haven't tested it yet, but it should be pretty darn close to working.)

The only trouble is the assumption that all values that are possibly pointers are in fact pointers, but other than that, it seems straightforward.

Answer 2

This problem is at least a subset of a research problem in its full generality, which suggests that there is not going to be an off-the-shelf system for you. A paper on the topic of record and replay of API calls appeared just two years ago in one of the premier operating systems conferences (OSDI 2008). A look on their website shows no available code -- nonetheless, the code, if available, may not be production-quality.

R2: An Application-Level Kernel for Record and Replay Zhenyu Guo, Xi Wang, Jian Tang, Xuezheng Liu, Zhilei Xu, Ming Wu, M. Frans Kaashoek, and Zheng Zhang, OSDI 2008

http://www.usenix.org/event/osdi08/tech/full_papers/guo/guo_html/

http://research.microsoft.com/en-us/people/zhenyug/r2-osdi08.pdf

Abstract

Library-based record and replay tools aim to reproduce an application's execution by recording the results of selected functions in a log and during replay returning the results from the log rather than executing the functions. These tools must ensure that a replay run is identical to the record run. The challenge in doing so is that only invocations of a function by the application should be recorded, recording the side effects of a function call can be difficult, and not executing function calls during replay, multithreading, and the presence of the tool may change the application's behavior from recording to replay. These problems have limited the use of such tools.

R2 allows developers to choose functions that can be recorded and replayed correctly. Developers annotate the chosen functions with simple keywords so that R2 can handle calls with side effects and multithreading. R2 generates code for record and replay from templates, allowing developers to avoid implementing stubs for hundreds of functions manually. To track whether an invocation is on behalf of the application or the implementation of a selected function, R2 maintains a mode bit, which stubs save and restore.

We have implemented R2 on Windows and annotated large parts (1,300 functions) of the Win32 API, and two higher-level interfaces (MPI and SQLite). R2 can replay multithreaded web and database servers that previous library-based tools cannot replay. By allowing developers to choose high-level interfaces, R2 can also keep recording overhead small; experiments show that its recording overhead for Apache is approximately 10%, that recording and replaying at the SQLite interface can reduce the log size up to 99% (compared to doing so at the Win32 API), and that using optimization annotations for BitTorrent and MPI applications achieves log size reduction ranging from 13.7% to 99.4%.