Fast way to trim lines at the beginning of a log file on iOS

Question

I'm looking for a fast, optimized way to trim log files on iOS. I want to specify that my log files have a maximum number of lines (e.g., 10,000). Appending new lines to the end of a text file seems relatively simple. However, I haven't yet found a fast way to trim lines at the beginning of the file. Here's the (slow) code I came up with.

    guard let fileURL = self.fileURL else {
        return
    }

    guard let path = fileURL.path else {
        return
    }

    guard let fileHandle = NSFileHandle(forUpdatingAtPath: path) else {
        return
    }

    fileHandle.seekToEndOfFile()
    fileHandle.writeData(message.dataUsingEncoding(NSUTF8StringEncoding)!)
    fileHandle.writeData("\n".dataUsingEncoding(NSUTF8StringEncoding)!)

    currentLineCount += 1

    // TODO: This could probably use some major optimization
    if currentLineCount >= maxLineCount {
        if let fileString = try? NSString(contentsOfURL: fileURL, encoding: NSUTF8StringEncoding) {
            var lines = fileString.componentsSeparatedByCharactersInSet(NSCharacterSet.newlineCharacterSet())
            lines.removeFirst()
            let newData = lines.joinWithSeparator("\n")
            fileHandle.seekToFileOffset(0)
            fileHandle.writeData(newData.dataUsingEncoding(NSUTF8StringEncoding)!)
        }
    }

    fileHandle.closeFile()

Answer 1

There are two aspects of your question. First, your code removes a single line from the log file. Therefore, once the limit is reached, every new log message causes the entire file to be read, shortened, and be re-written.

It would be more effective to use a "high-water mark" and a "low-water mark". For example, if you want the last 10.000 lines to be preserved, let the log file grow until it has 15.000 lines, and only then truncate it to 10.000 lines. This reduces the number of "trim actions" considerably.

The second part is about the truncating itself. Your code loads the file into an NSString, which requires the conversion of UTF-8 data to Unicode characters (and fails if there is a single invalid byte in the log file). Then the string is split into an array, one array element removed, the array concatenated to a string again, and then written back to the file, which converts the Unicode characters to UTF-8.

I haven't done performance tests, but I can imagine that it could be faster to operate on binary data only, without the conversions to NSString, Array and back. Here is a possible implementation which removes a given number of lines from the start of a file:

func removeLinesFromFile(fileURL: NSURL, numLines: Int) {

    do {
        let data = try NSData(contentsOfURL: fileURL, options: .DataReadingMappedIfSafe)
        let nl = "\n".dataUsingEncoding(NSUTF8StringEncoding)!

        var lineNo = 0
        var pos = 0
        while lineNo < numLines {
            // Find next newline character:
            let range = data.rangeOfData(nl, options: [], range: NSMakeRange(pos, data.length - pos))
            if range.location == NSNotFound {
                return // File has less than `numLines` lines.
            }
            lineNo++
            pos = range.location + range.length
        }

        // Now `pos` is the position where line number `numLines` begins.
        let trimmedData = data.subdataWithRange(NSMakeRange(pos, data.length - pos))
        trimmedData.writeToURL(fileURL, atomically: true)

    } catch let error as NSError {
        print(error.localizedDescription)
    }
}

Answer 2

I have updated Martin R answer to Swift 3, and I also changed it so that we can pass the number of lines to keep instead of the number of lines to remove:

func removeLinesFromFile(fileURL: URL, linesToKeep numLines: Int) {

    do {
        let data = try Data(contentsOf: fileURL, options: .dataReadingMapped)
        let nl = "\n".data(using: String.Encoding.utf8)!

        var lineNo = 0
        var pos = data.count-1
        while lineNo <= numLines {
            // Find next newline character:
            guard let range = data.range(of: nl, options: [ .backwards ], in: 0..<pos) else {
                return // File has less than `numLines` lines.
            }
            lineNo += 1
            pos = range.lowerBound
        }

        let trimmedData = data.subdata(in: pos..<data.count)
        try trimmedData.write(to: fileURL)

    } catch let error as NSError {
        print(error.localizedDescription)
    }
}