Ruby will open file but not write to it?

Question

I'm trying to create a basic ruby scraper that will grab all words 8 letters or longer from html source code. Then it saves these in a file corresponding to the first character of the word. Seems simple huh?

    re = /\w{8,}/
    cre = /[a-z0-9]/
    a = b.html    #This grabs the html from the browser
    matchx = a.scan(re)
    matchx.each do |xx|
        word = xx.to_s.downcase.chomp
        fchar = word[0].chr

        if (fchar.match(cre)) #Not sure if I need this
            @pcount += 1
            fname = @WordsFName+fchar   #@WordsFName is a prefix
            tmpF = File.open(fname,"a+")

            #Check for duplicates, if not write to file
            exists = File.readlines(fname).any? { |li| li[word] }
            if (!exists)                    
                tmpF.write(word+"\n")
                print word 
                @wcount += 1
            end
        end

    end

Ruby successfully grabs all the words, gets the first character, and opens all the necessary files, but fails to write to it. Also, the print method prints all words including duplicates, but inspecting the any? method on irb gave no problems..

Answer 1

File#write is buffered and you do nothing to flush or close tmpF between your write and the File.readlines(fname), so the readlines will never see the output until it's flushed. I don't see any call to close on tmpF so, it's not clear when the write data will get flushed except program exit when the file object is finalized, or GC some time after tmpF goes out of scope.

You could manually flush after the write with tmpF.flush, or make that default behavior with tmpF.sync = true after the open.

Note that as each file gets bigger, the cost of your dup check is going to balloon as it rereads the whole file. If the word set fits in memory, consider just keeping a a hash of words you've seen, if it's bigger than can be stored in memory, consider a key-value store instead of rereading a serial file every time.

I played around in irb to understand flushing behavior. The main problem with OP code is there's no explicit/implicit flush or close on the tmpF file. So the partial writes which are likely less than the buffer size only get written when the tmpF File object gets garbage collected or upon program exit. tmpF gets assigned a newly opened file object each time through the loop, so the files opened on prior iterations only get flushed when they get finalized at GC.

irb(main):001:0> t=File.open('zzz','a+')
=> #<File:zzz>
irb(main):002:0> t.write '123'
=> 3
irb(main):003:0> File.readlines('zzz')
=> []
irb(main):004:0> t=File.open('zzz','a+')
=> #<File:zzz>
irb(main):005:0> t.write '456'
=> 3
irb(main):006:0> File.readlines('zzz')
=> []
irb(main):007:0> t.close
=> nil
irb(main):008:0> File.readlines('zzz')
=> ["456"]
irb(main):009:0> t=File.open('zzz','a+')
=> #<File:zzz>
irb(main):010:0> t.write '789'
=> 3
irb(main):011:0> File.readlines('zzz')
=> ["456"]
irb(main):012:0> t.flush
=> #<File:zzz>
irb(main):013:0> File.readlines('zzz')
=> ["456789"]
irb(main):014:0> GC.start
=> nil
irb(main):015:0> File.readlines('zzz')
=> ["456789123"]