displaying a large amount of formatted text in Python

Question

I have two large identical-sized files. One is ASCII plain text, and the other is a colour-coded overlay, one byte per text character in the corresponding file.

These files can be large - upto 2.5 MB; possibly substantially more, perhaps over 100MB later.

I want to display the text is a scrollable text viewer, using the second file as the overlay. Looking something like this:

Tkinter Text window with tagged text for colours http://img713.imageshack.us/img713/2584/statsviewer01d.png

So I made a simple tkinter Text widget window in Python with scrollbar and such.

My code looks like this:

hottest = 0
for heat in heatmap:
    hottest = max(hottest,ord(heat))
hottest += 1
for heat in xrange(0,hottest):
    factor = int((float(heat)/float(hottest))*100.0)
    # an observation; tkinter seems to normalise custom colours
    # to nearest in it's palette, which means I can't use custom
    # gradients of red; if anyone knows how to use subtle custom colours?
    bgcolour = "gray%d" % (100-factor)
    fgcolour = "gray%d" % factor
    text.tag_config("n%d"%heat,background=bgcolour,foreground=fgcolour)
text.insert("1.0",f.read())
ofs = 0
for heat in heatmap:
    if 0 != ord(heat):
        coord_start = "1.0 + %d chars"%ofs
        coord_stop = "1.0 + %d chars"%(ofs+1)
        text.tag_add("n%d"%ord(heat),coord_start,coord_stop)
    ofs += 1
text.config(state=DISABLED)
text.focus()

However, I run into horrid performance problems:

1. loading the text

   • if I simply use textwidget.text = textfile.read() it loads instantly, even for the large file; but then the problem is colour-coding it. (see code above). To add the tags, one character at a time, it seems I force it to take a painter algorithm; by the time the file is about 300KB in size, I give up waiting for it to load
   • if I read the next character from the textfile and the colourfile, and append a properly tagged character, it takes forever; a file bigger than about 200KB and I give up waiting for it to load

2. Scrolling. If this much formatting visible in the current window, it goes very slowly

If I was approaching this problem in, say, Dephi or wxWidgets or whatever, I'd have a custom-drawn control.

What's the most straightforward way in Python and Tkinter?

Answer 1

How about using a scrollable canvas instead, and only ever drawing the text/heatmap that is exposed by the user? That should give you a quick initial draw and a quick redraw when things move around, regardless of the size of the file.

If you want more speed and more control, then you would need some sort of virtual canvas where only the area on display and an area around it actually exists, anything else is only drawn as and when it gets referenced. I don't think TkInter gives you that much control, although things like the Widget Construction Kit (WCK) should do.