How can I handle a list with several hundred items in HTML? (Using Javascript)

Question

I am creating a web page which is designed for mobile safari, and I create a list dynamically. The list is created from a text file in which I receive information and I append this information to the list. It works fine when there are about 200 items, but when the file is very large, (I have tried with up to 4000 items) the page becomes very slow, scrolling and selecting this items is very hard. I know I should not create so much HTML elements, but I am looking for a way to create a shorter list and replace the information on the list's elements depending on how much scrolling you have done. Any Ideas?

Answer 1

Clarification

To keep things simple I will consume right from the start the possibility of using a specialized JavaScript UI library. Personally I would not use such a solution since getting involved with a library creates additional constrains (and possibly bloat) to your project (unless you are working on a project that relies heavily on client-side UI components, in which case you should choose one UI library and stick with it).

Also, I will not consider "scrolling" as a solution (i.e. creating an endless list of DOM objects and simply scrolling through them). IMHO that is not really a "solution" to the problem, it is simply a method of generating the problem that led you here.

Think flow

You have to keep in mind that on the client side, the JavaScript language is in a sense "attached" to the browser's Document Object model. Your JavaScript code is practically "telling" the browser to modify the DOM hierarchy in a myriad of ways, and then the browser conforms. This is the main reason why pure JavaScript code tends to run fast and then things slow down dramatically when starting to manipulate DOM objects.

The main rule to follow, in my opinion, is to minimize as much as humanly possible the amount of DOM object manipulation by following this flow:

1. If possible, do it server-side: this can prove to be a very consistent and reliable solution; if your project allows for this, you should only send reasonably-sized segments of the data to the browser at any single time (e.g. one page from the entire set), this will allow the client to properly display and manipulate that data in a speedy and responsive way.

2. If you must do it on the client-side, avoid touching the DOM as much as possible: this is essential; if you receive 4000 objects in a JS script, keep them in memory, don't rush to translate all those elements into DOM objects; that's a death sentence.

3. Only manipulate the DOM at the last possible moment and touching the least amount of objects: since modifying the DOM takes a lot of effort (compared with other in-memory operations). Another danger here is memory leaking (browsers aren't perfect), since when creating and discarding many DOM objects, some of those discarded object might still survive in memory, thus producing a memory leak (which could eventually hog the browser or even the system).

Solutions

These are the approaches on the table:

1. Server-side pagination: perhaps this is obvious, but too many times people try to overlook this option, and I think that's wrong. Again, if your projects allows this, it is very difficult for such a solution to fail you: it can be very consistent and reliable (which is why it is a classic).

2. Client-side pagination: Obviously you could send the entire data to the browser, handle it in JS and then display it in reasonably-sized chunks at once, never exceeding a certain amount of generated DOM objects. This might seem attractive and more simple than Solution 1, however, beware that if your data is bidirectional (i.e. implies intervention and feedback from the user that must return to the server) and important (e.g. more important that statistical/reporting data that only needs to be displayed), you should probably avoid this option.

   The reason is that even if JavaScript is fast and efficient enough to juggle this data in memory, it is not as safe; you never know what might happen client-side: the memory could get corrupted, the browser might crash etc.; basically you have no reasonable guarantee for your data. A server is better prepared to handle important data. What's more, it will be very difficult to have history navigation between pages and URL addresses that go to specific pages within the data-set (both are possible but not without a headache anyway).

3. Client-side dynamic scrolling: instead of displaying the objects from JS memory as full pages, you simply show a reasonable sub-set, let's say 100 items and then scroll down by taking one object (or more) from the top and moving it to the bottom of the list, altering it's contents accordingly. Obviously this option presents the same dangers as Solution 2, but it is still a very good solution.

Example

Considering that your projects works for mobile devices I consider the last approach as probably more feasible to you, so here is a (very) simplistic example of how it might be done using MooTools (obviously the principle can be applied using any framework):

<html>

<head>

    <title>Endless scrolling example.</title>

    <!-- Include the MooTools framework. -->
    <script type="text/javascript" src="mootools-1.2.4-core-yc.js"></script>

    <!-- Note: we'll use id's to select objects since it's faster. -->

</head>

<body>

    <!-- Scroll up. -->
    <a href="javascript:void(0);" id="list_up_button">Up!</a>

    <!-- The list (i.e. container). -->
    <ul id="list_container">
    </ul>

    <!-- Scroll down. -->
    <a href="javascript:void(0);" id="list_down_button">Down!</a>

</body>

<!-- Our script. -->
<script type="text/javascript">

    window.addEvent('domready', function() {

        // database
        var list = {};

        // options
        var list_size = 5000;   // total list size
        var list_offset = 0;    // initial list position
        var list_subset = 40;   // the smount of displayed items
        var scroll_step = 10;    // items to scroll in one step
        var time_delay = 50;    // time delay between scroll steps

        // make dummy items
        for (var i = 0; i < list_size; i++) {
            var red = Math.floor(i * 2) % 256;
            var green = Math.floor(i * 3) % 256;
            var blue = Math.floor(i * 4) % 256;

            list[i] = "<span style=\"color:rgb(" + red + ", " + green + ", " + blue + ");\">" +
                      "My name is 'Object no." + (i + 1) + "'." +
                      "</span>";
        }

        // get container
        var list_container = $('list_container')

        // generate DOM objects
        for (var i = 0; i < list_subset; i++) {
            list_container.grab(new Element('li', { html: list[i] }));
        }

        // Up scroller.
        function up() {

            // check end
            if (list_offset <= 0) {
                return false;
            }

            // get element
            var element = list_container.getLast();

            // devance offset
            list_offset--;

            // re-write element
            element.set('html', list[list_offset]);

            // move top element to top
            list_container.grab(element, 'top');

            // success
            return true;
        }

        // Down scroller.
        function down() {

            // check end
            if (list_offset >= list_size - list_subset) {
                return false;
            }

            // get element
            var element = list_container.getFirst();

            // advance offset
            list_offset++;

            // re-write element
            element.set('html', list[list_offset + list_subset - 1]);

            // move top element to bottom
            list_container.grab(element, 'bottom');

            // success
            return true;
        }

        // Repeater function.
        function repeater(direction) {

            for (var i = 0; i < scroll_step; i++) {

                // scroll
                if (direction() == false) {

                    // deactivate repeater
                    $clear(list.scroll_period);
                }
            }

        }

        // configure up scroll
        $('list_up_button').addEvents({

            // activate scroll
            'mouseenter': function() {
                list.scroll_period = repeater.periodical(time_delay, null, up);
            },

            // deactivate scroll
            'mouseleave': function() {
                $clear(list.scroll_period);
            }
        });

        // configure up scroll
        $('list_down_button').addEvents({

            // activate scroll
            'mouseenter': function() {
                list.scroll_period = repeater.periodical(time_delay, null, down);
            },

            // deactivate scroll
            'mouseleave': function() {
                $clear(list.scroll_period);
            }
        });

    });

</script>

Sorry for the rather long code... hope it helps.