Dynamically displaying large amounts of text by partially displaying data in textarea

Question

The Goal:

Optimize a web page that needs to display large amount of text data (25mb+) in a textarea or content editable div, without losing too much performance.

Currently, loading a 10mb file takes ~3 seconds on Chrome. I would like this to be 1 second max.

---

Ideas and Attempts:

I am loading local text files from a users computer using <input type="file"> and have no problem loading large files directly into memory. However as soon as I try and display this text data in a textarea, I naturally run into performance issues.

I have spellcheck, auto capitalization, and auto complete all disabled, and this certainly helped, however I would like to minimize the amount of lag when attempting to render large files (files greater than 10mb, max of 100mb would be ideal).

<textarea autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false">

One idea I had was to only render say 100 lines before and 100 lines after the current line, and when the users scrolls the textarea, I would then just switch out what data is being displayed. I can swap a few hundred lines without any noticeable lag, but hundreds of thousands locks up the entire page.

I was also looking at projects such as CodeMirror which is used in some javascript based text editors, and chrome dev tools. However a quick test showed similar performance issues when initially loading large amounts of text.

Another idea was to use highlight.js to render the text dom elements, but I was also noticing large amounts of large when dealing with thousands of DOM elemetns.

This site seems to tackle a similar example by dynamically creating and displaying the dom elements, instead of attempting to render all at once.

I have found that if the number of records in the grid becomes more then just a few thousands the grid gets very slow because the rendering speed is directly related to the number of nodes in the DOM. If the number of nodes in the DOM is more then 40-50k (depending on your computer configuration and amount of memory), your browser will crash or will become unresponsive.

So, I decided to set out on a quest to do two things: (1) dynamically create records as user scrolls (2) optimize grid to handle large data sets. After a few weeks of work, the grid was optimized and ready for testing.

I think this is similar to my first idea, but I have not tried this approach yet.

I'm hoping someone who's had experience with something similar might be able to offer some advice on which path to take, or offer some additional ideas.

Before anyone asks, I cannot not show this data, it needs to be user editable, it does not need to highlight code, or show line numbers. Lastly, the entire text file is being loaded with a FileReader into a variable.

I would like to avoid uploading the file to my webserver if possible for end user privacy and NDA concerns.

Server config: Ubuntu 16.04 LAPP Stack with Laravel 5.4, open to NodeJS solutions though. Use of jQuery is allowed.

---

Proposed Solution(s):

Lazy Loading - only displaying say 300 line "chunks" at a time as the user scrolls. Would need to make the scrollbar the appropriate height ahead of time in this case though. - Also, should unload these "chunks" as the users scrolls to reduce total the DOM rendering load.

Pseduo code:

c_chunk = scrollpos / scrollheight * totalChunks;
chunk_block = chunk[c_chunk--] + chunk[c_chunk] + chunk[c_chunk++];
my_textarea.val(chunk_block);

Any thoughts on this method?

Thanks all.

Answer 1

As promised here's my solution:

So after fumbling around with various ideas and trying different methods, I think I finally settled on using Ace for my project. I chose Ace for a few reasons.

1. Well documented API that was easy to understand in addition to a fairly large and active community.
2. Performance, Ace hands down gets far superior performance compared to my test with Code Mirror using large files. (500,000 lines, github says Ace has been tested with up to 4 million lines.
3. Features, out of the box Ace has more features to get you up and running with little user setup or configuration. Getting started took less than 10 lines total!
4. It doesn't support as many browsers as Code Mirror, but my project is running with WebGL so I was not very concerned.

So you understand my reasoning, and you want to get Ace up and running yourself? Easy, just hop on over to the build repo and grab one whichever flavor suites your needs. I grabbed the src-min release as I will just add it to my build script.

Then just include the javascript file in whatever way you do:

<script src="/ace/ace.js" type="text/javascript" charset="utf-8"></script>

And an element to your page with id="editor". In my case I'll attach it directly to a div:

<div id="editor">Any text you want to have auto displayed</div>

and in a javascript file:

var editor = ace.edit("editor");
editor.setTheme("ace/theme/chrome");
editor.session.setMode("ace/mode/javascript");
editor.$blockScrolling = Infinity;

If you just want to look around to see what languages/themes/options/etc. are available, just head over to Ace's kitchen sink page and play around with the editor.

Now I also needed a few functions, such as being able to load a file into the text area - and you can do that with the following:

Load local file:

// Add event listener to the file input
// Note: This will not fire twice if the user opens File A, then re-opens file A
// To detect re-opening a file you will need to clear the input
// after the file is read so that it "changes" upon re-opening
document.getElementById('open-file').addEventListener('change', handleFileOpen, false);

// First we create the function handler that fires when our input is changed
function handleFileOpen(e) {
    var file = e.target.files[0]; // Get first file selected

    // Load file using file reader function
    loadFile(file);
}

function loadFile(file) {    
    // Create file reader and relevant Ace code
    var reader = new FileReader();
    reader.onload = function(e) {
        // Get text contents of file
        var data = e.target.result;

        // Update Ace Editor with loaded data
        editor.setValue(data, -1);
    };

    // Now that we defined the function we want to run, read the file
    reader.readAsText(file);
}

The -1 in the setValue() function means place cursor at the beginning of the file (top) - 1 would be the bottom.

There are tons of events and properties you can hook into and mess in order to tweak this editor to your exact liking. They are all extremely straight forward and easy to use so if you're undecided, it's worth the time to try it out.

I wasted 2 days trying to get clusterize + highlightjs to work with my setup and gave up finally and replaced that entire setup with Ace in less than a day. Really impressed with editor so far!

Here's a link to their "How-to Guide": https://ace.c9.io/#nav=howto
and their API reference (which is extremely useful): https://ace.c9.io/#nav=api