Paginating chronologically prioritized Firebase children

Question

tl;dr Performing basic pagination in Firebase via startAt, endAt and limit is terribly complicated, there must be an easier way.

I'm constructing an administration interface for a large number of user submissions. My initial (and current) idea is to simply fetch everything and perform pagination on the client. There is however a noticable delay when fetching 2000+ records (containing 5-6 small number/string fields each) which I attribute to a data payload of over 1.5mb.

Currently all entries are added via push but I'm a bit lost as to how paginate through the huge list.

To fetch the first page of data I'm using endAt with a limit of 5:

ref.endAt().limit(10).on('child_added', function(snapshot) {
    console.log(snapshot.name(), snapshot.val().name)
})

which results in the following:

-IlNo79stfiYZ61fFkx3 #46 John
-IlNo7AmMk0iXp98oKh5 #47 Robert
-IlNo7BeDXbEe7rB6IQ3 #48 Andrew
-IlNo7CX-WzM0caCS0Xp #49 Frank
-IlNo7DNA0SzEe8Sua16 #50 Jimmmy

Firstly to figure out how many pages there are I am keeping a separate counter that's updated whenever someone adds or removes a record.

Secondly since I'm using push I have no way of navigating to a specific page since I don't know the name of the last record for a specific page meaning an interface like this is not currently possible:

To make it simpler I decided on simply having next/previous buttons, this however also presents a new problem; if I use the name of the first record in the previous result-set I can paginate to the next page using the following:

ref.endAt(null, '-IlNo79stfiYZ61fFkx3').limit(5).on('child_added', function(snapshot) {
    console.log(snapshot.name(), snapshot.val().name)
})

The result of this operation is as follows:

-IlNo76KDsN53rB1xb-K #42 William
-IlNo77CtgQvjuonF2nH #43 Christian
-IlNo7857XWfMipCa8bv #44 Jim
-IlNo78z11Bkj-XJjbg_ #45 Richard
-IlNo79stfiYZ61fFkx3 #46 John

Now I have the next page except it's shifted one position meaning I have to adjust my limit and ignore the last record.

To move back one page I'll have to keep a separate list on the client of every record I've received so far and figure out what name to pass to startAt.

Is there an easier way of doing this or should I just go back to fetching everything?

Answer 1

We're working on adding an "offset()" query to allow for easy pagination. We'll also be adding a special endpoint to allow you to read the number of children at a location without actually loading them from the server.

Both of these are going to take a bit though. In the meantime the method you describe (or doing it all on the client) are probably your best bet.

If you have a data structure that is append-only, you could potentially also do pagination when you write the data. For example: put the first 50 in /page1, put the second 50 in /page2, etc.

Answer 2

Another way to accomplish this is with two trees:

Tree of ids: { id1: id1, id2: id2, id3: id3 }

Tree of data: { id1: ..., id2: ..., id3: ... }

You can then load the entire tree of ids (or a big chunk of it) to the client, and do fancy pagination with that tree of ids.

Answer 3

Here's a hack for paginate in each direction.

// get 1-5
ref.startAt().limit(5)

// get 6-10 from 5
ref.startAt(null, '5th-firebase-id' + 1).limit(5)

// get 11-15 from 10
ref.startAt(null, '10th-firebase-id' + 1).limit(5)

Basically it's a hack for startAtExclusive(). You can anything to the end of the id.

Also figured out endAtExclusive() for going backwards.

// get 6-10 from 11
ref.endAt(null, '11th-firebase-id'.slice(0, -1)).limit(5)... 

// get 1-5 from 6
ref.endAt(null, '6th-firebase-id'.slice(0, -1)).limit(5)... 

Will play with this some more but seems to work with push ids. Replace limit with limitToFirst or limitToLast if using firebase queries.