High Performance Event Log

Question

So I've been trying various ways to get Event Log data in bulk (1000+ records/second).

I need something that can filter out old logs, right now I store the last recorded event record ID and retrieve all records where the event ID is greater than that....

I've tried EventLogQuery/EventLogReader, this works fast except when I want to pull message data, in order to get a formatted message for security logs I need to call EventLogRecord.FormattedMessage(), this brings my log speed to about 150/second with easy to format logs, even worse with complicated ones.

I've tried System.Diagnoistics.EventLog, this doesn't allow me to build filters, so every time I run this it must load ALL event logs, then I can parse off any duplicates (from the last scan). I have a sever that has 200k event logs over the past two days, memory usage gets terrible due to this, so that is a no-go.

I've tried WMI using System.Management.ManagementObjectCollection, this has filtering and can pull message data from the security event log FAST (approaching ~1000/second), however it will go to about 50/60k and start to drag it's feet, down to doing about 1-2/second, eventually I'll get a Quota Violation error. :(

So either:

Is there a way to avoid the quota violation error, or do I want to use some other method for pulling event logs at this speed?

Edit:

I wrote a blog post detailing what I've learned about this:

http://www.roushtech.net/2013/10/30/high-performance-event-log-reading/

Mostly: WINAPI is your best bet, either write C++/CLR or use PInvoke.

Answer 1

I might look into the command-line utility:

C:/> Wevtutil.exe qe Application /f:XML

Documentation: http://technet.microsoft.com/en-us/library/cc732848(WS.10).aspx

It will save a bookmark and continue from bookmark as well as serve to process much more complicated queries.

For example, let's say you want all events of id 1530, in a group of 1000, from the last bookmark you saved, pre-rendered in english:

Wevtutil qe Application /f:RenderedXml /bm:bookmark.pos /sbm:bookmark.pos /c:1000 /e:Log /l:en-us /q:Event[System/EventID='1530']

This yeilds events that look like:

<Log>
  <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    <System>
      <Provider Name="Microsoft-Windows-User Profiles Service" Guid="{89B1E9F0-5AFF-44A6-9B44-0A07A7CE5845}"/>
      <EventID>1530</EventID>
      <Version>0</Version>
      <Level>3</Level>
      <Task>0</Task>
      <Opcode>0</Opcode>
      <Keywords>0x8000000000000000</Keywords>
      <TimeCreated SystemTime="2011-06-16T22:55:32.305140500Z"/>
      <EventRecordID>26196</EventRecordID>
      <Correlation/>
      <Execution ThreadID="4168" ProcessID="1660"/>
      <Channel>Application</Channel>
      <Computer>hostname.WORKGROUP</Computer>
      <Security UserID="S-1-5-18"/>
    </System>
    <EventData Name="EVENT_HIVE_LEAK">
      <Data Name="Detail">0 user registry handles leaked from \Registry\User\S-1-5-82: </Data>
    </EventData>
    <RenderingInfo Culture="en-US">
      <Message>Windows detected your registry file is still in use by other applications or services. The file will be unloaded now. The applications or services that hold your registry file may not function properly afterwards. DETAIL - 0 user registry handles leaked from \Registry\User\S-1-5-82: </Message>
      <Level>Warning</Level>
      <Task/>
      <Opcode>Info</Opcode>
      <Channel>Application</Channel>
      <Provider>Microsoft-Windows-User Profile Service</Provider>
      <Keywords/>
    </RenderingInfo>
  </Event>

Using a fast light-weight xml parser, you should be able to scream through this data while in parallel producing the next result page. It's not even necessary to write this to disk if you capture the process output directly. Just be sure you are also capturing the std::error since this utility will also write errors to that output.

Answer 2

I found that managed code was too slow and I ended up using the win32 APIs to retrieve the event logs (local and/or remote). They are easy enough to implement and there are a lot of examples out there of how to do it. It will also be much faster than anything you try and do with xml.

New API: http://msdn.microsoft.com/en-us/library/aa385780(v=vs.85).aspx

Old API: http://msdn.microsoft.com/en-us/library/aa363652(v=vs.85).aspx

I also found that the pre-vista API worked fine for what I needed and actually performed better than the new API on post-vista machines.