GraphQL Dataloader not knowing keys in advance

Question

Dataloader is able to batch and cache requests, but it can only be used by either calling load(key) or loadMany(keys).

The problem I am having is that sometimes I do not know they keys of the items I want to load in advance.

I am using an sql database and this works fine when the current object has a foreign key from a belongsTo relation with another model.

For example a user that belongs to a group and so has a groupId. To resolve the group you would just call groupLoader.load(groupId).

On the other hand, if I wanted to resolve the users within a group, of which there could be many I would want a query such as

SELECT * from users where user.groupId = theParticularGroupId

but a query such as this doesn't use the keys of the users and so I am not sure how make use of dataloader.

I could do another request to get the keys like

SELECT id from users where user.groupId = theParticularGroupId

and then call loadMany with those keys... But I could have just requested the data directly instead.

I noticed that dataloader has a prime(key, value) function which can be used to prime the cache, however that can only be done once the data is already fetched. At which point many queries would already have been sent, and duplicate data could have been fetched.

---

Another example would be the following query

query {
  groups(limit: 10) {
    id
    ...
    users {
      id
      name
      ...
    }
  }
}

I cannot know the keys if I am searching for say the first or last 10 groups. Then once I have these 10 groups. I cannot know the keys of their users, and if each resolver would resolve the users using a query such as

SELECT * from users where user.groupId = theParticularGroupId

that query will be executed 10 times. Once the data is loaded I could now prime the cache, but the 10 requests have already been made.

Is there any way around this issue? Perhaps a different pattern or database structure or maybe dataloader isn't even the right solution.

Answer 1

You'll want a dataloader instance for the lookup you can do, in this case you have a group ID and you want the users:

import DataLoader from 'dataloader';

const userIdsForGroupLoader = new DataLoader(groupIds => batchGetUsersIdsForGroups(groupIds));

Now your batchGetUsersForGroups function is essentially has to convert an array of group IDs to an array of arrays of users (one array of user IDs for each group).

You'd start off with an IN query:

SELECT id from users where user.groupId in (...groupIds)

This will give you a single result set of users, which you'll have to manipulate, by grouping them by their groupId, the array should be ordered according to the original array of groupIds. Make sure you return an empty array for groupIds that don't have any users.

Note that in this we're only returning the user ids, but you can batch fetch the users in one go once you have them. You could tweak it slightly to return the users themselves, you'll have to decide for yourself if that's the right approach.

Everything I mention in this article can be achieved using clever use of Dataloader. But the key takeaway is that the values you pass to the load/loadMany functions don't have to correspond to the IDs of the objects you're trying to return.