Reading file content changes in .NET

Question

In Linux, a lot of IPC is done by appending to a file in 1 process and reading the new content from another process.

I want to do the above in Windows/.NET (Too messy to use normal IPC such as pipes). I'm appending to a file from a Python process, and I want to read the changes and ONLY the changes each time FileSystemWatcher reports an event. I do not want to read the entire file content into memory each time I'm looking for changes (the file will be huge)

Each append operation appends a row of data that starts with a unique incrementing counter (timestamp+key) and ends with a newline.

Answer 1

You can store the offset of the last read operation and seek the file to that offset when you get a changed file notification. An example follows:

Main method:

public static void Main(string[] args)
{
    File.WriteAllLines("test.txt", new string[] { });

    new Thread(() => ReadFromFile()).Start();

    WriteToFile();
}

Read from file method:

private static void ReadFromFile()
{
    long offset = 0;

    FileSystemWatcher fsw = new FileSystemWatcher
    {
        Path = Environment.CurrentDirectory,
        Filter = "test.txt"
    };

    FileStream file = File.Open(
        "test.txt",
        FileMode.Open,
        FileAccess.Read,
        FileShare.Write);

    StreamReader reader = new StreamReader(file);
    while (true)
    {
        fsw.WaitForChanged(WatcherChangeTypes.Changed);

        file.Seek(offset, SeekOrigin.Begin);
        if (!reader.EndOfStream)
        {
            do
            {
                Console.WriteLine(reader.ReadLine());
            } while (!reader.EndOfStream);

            offset = file.Position;
        }
    }
}

Write to file method:

private static void WriteToFile()
{
    for (int i = 0; i < 100; i++)
    {
        FileStream writeFile = File.Open(
            "test.txt",
            FileMode.Append,
            FileAccess.Write,
            FileShare.Read);

        using (FileStream file = writeFile)
        {
            using (StreamWriter sw = new StreamWriter(file))
            {
                sw.WriteLine(i);
                Thread.Sleep(100);
            }
        }
    }
}

Answer 2

    using (FileStream fs = new FileStream
       (fileName, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
    {
        using (StreamReader sr = new StreamReader(fs))
        {
            while (someCondition)
            {
                while (!sr.EndOfStream)
                    ProcessLinr(sr.ReadLine());
                while (sr.EndOfStream)
                    Thread.Sleep(100);
                ProcessLinr(sr.ReadLine());            
            }
        }
    }

this will help you read only appended lines