How to get all documents from a collection in FaunaDB?

Question

I already have an answer:

const faunadb = require('faunadb')
const q = faunadb.query

exports.handler = async (event, context) => {
  const client = new faunadb.Client({
    secret: process.env.FAUNADB_SERVER_SECRET
  }) 

  try {  
    // Getting the refs with a first query
    let refs = await client.query(q.Paginate(q.Match(q.Index('skus'))))
    // Forging a second query with the retrieved refs
    const bigQuery = refs.data.map((ref) => q.Get(ref))
    // Sending over that second query
    let allDocuments = await client.query(bigQuery)
    // All my documents are here!
    console.log('@allDocuments: ', allDocuments);
    //...
  } catch (err) {
    // ...
  }
}

But I find it unsatisfying because I'm making 2 queries for what seems like one the most trivial DB call. It seems inefficient and wordy to me.

As I'm just learning about FaunaDB, there's probably something I don't grasp here. My question could be split into 3:

• Can I query for all documents in a single call?
• If not, why not? What's the logic behind such a design?
• Could I make such a query without an index?

Answer 1

FaunaDB's FQL language is quite similar to JavaScript (which helps a lot if you want to do conditional transactions etc).

In essence, FaunaDB also has a Map. Given that your index contains only one value that is the reference you can write this:

q.Map(
  q.Paginate(q.Match(q.Index('skus'))),
  q.Lambda(x => q.Get(x))
)

For this specific case, you actually do not need an index since each collection has a built-in default index to do a select all via the 'Documents' function.

q.Map(
  q.Paginate(q.Documents(q.Collection('<your collection>'))),
  q.Lambda(x => q.Get(x))
)

Now in case the index that you are using returns multiple values (because you would want to sort on something other than 'ref') then you need to provide the same amount of parameters to the Lambda as the amount of values that were defined in the index. Let's say my index has ts and ref in values because I want to sort them on time, then the query to get all values becomes:

q.Map(
  q.Paginate(q.Match(q.Index('<your index with ts and ref values>'))),
  q.Lambda((ts, ref) => q.Get(ref))
)

Values are used for range queries/sorting but also define what the index returns

Coming back to your questions:

- Can I query for all documents in a single call?

Absolutely, I would advice you to do so. Note that the documents you will get are paginated automatically. You can set the page size by providing a parameter to paginate and will get back an 'after' or 'before' attribute in case the page is bigger. That after or before can be again presented to the Paginate function as a parameter to get a next or previous page: https://docs.fauna.com/fauna/current/api/fql/functions/paginate

- Could I make such a query without an index?

No, but you can use the built-in index as explained above. FaunaDB protects users from querying without an index. Since it is a scalable database that could contain massive data and is pay-as-you-go it's a good idea to prevent users from shooting themselves in the foot :). Pagination and mandatory Indexes help to do that.

As to the why FQL is different. FQL is a language that is not declarative like many querying languages. Instead it's procedural, you write exactly how you fetch data. That has advantages:

1. By writing how data is retrieved you can exactly predict how a query behaves which is nice-to-have in a pay-as-you-go system.
2. The same language can be used for security rules or complex conditional transactions (update certain entities or many entities ranging over different collections depending on certain conditions). It's quite common in Fauna to write a query that does many things in one transaction.
3. Our flavour of 'stored procedures' called User Defined Functions are just written in FQL and not another language.

Querying is also discussed in this tutorial that comes with code in a GitHub repository which might give you a more complete picture: https://css-tricks.com/rethinking-twitter-as-a-serverless-app/

Answer 2

Can I query for all documents in a single call?

Yes, if your collection is small. The Paginate function defaults to fetching 64 documents per page. You can adjust the page size up to 100,000 documents. If your collection has more than 100,000 documents, then you have to execute multiple queries, using cursors to fetch subsequent documents.

See the Pagination tutorial for details: https://docs.fauna.com/fauna/current/tutorials/indexes/pagination

If not, why not? What's the logic behind such a design?

For an SQL database, SELECT * FROM table is both convenient and, potentially, a resource nightmare. If the table contains billions of rows, attempting to serve results for that query could consume the available resources on the server and/or the client.

Fauna is a shared database resource. We want queries to perform well for any user with any database, and that requires that we put sensible limits on the number of documents involved in any single transaction.

Could I make such a query without an index?

No, and yes.

Retrieving multiple results from Fauna requires an index, unless you are independently tracking the references for documents. However, with the Documents function, Fauna maintains an internal index so you don't need to create your own index to access all documents in a collection.

See the Documents reference page for details: https://docs.fauna.com/fauna/current/api/fql/functions/documents

Returning to your example code, you are executing two queries, but they could easily be combined into one. FQL is highly composable. For example:

let allDocuments = await client.query(
  q.Map(
    q.Paginate(q.Documents(q.Collection("skus"))),
    q.Lambda("X", q.Get(q.Var("X")))
  )
)

Your observation that FQL is wordy, is correct. Many functional languages exhibit that wordiness. The advantage is that any functions that accept expressions can be composed at will. One of the best examples of composability, and how to manage inter-document references, is presented in our E-commerce tutorial, specifically, the section describing the submit_order function: https://docs.fauna.com/fauna/current/tutorials/ecommerce#function