RTK Query - Infinite Scrolling, retaining existing data

Question

I am attempting to implement infinite scrolling functionality in our current application;

We first fetch the first 5 'Posts' for a page. Upon scrolling to the bottom of the page, we then fetch the next 5 Posts.

This works nicely, however using the same query means that the existing data (the first 5 posts) has been replaced by the new data.

Is it possible to merge the existing data with the new data?

I could merge them in place, for example with something like; const posts = [newPosts, oldPosts] but then we lose the data invalidation provided by RTK Query if the existing data is modified.

What is the recommended approach for this case?

Answer 1

In RTK 1.9 it is now possible to use the merge option to merge newly fetched data with the data that currently lives inside the cache. Make sure you use the option together with serializeQueryArgs or forceRefetch to keep a cache entry for the data.

createApi({
  baseQuery: fetchBaseQuery({ baseUrl: '/' }),
  endpoints: (build) => ({
    listItems: build.query<string[], number>({
      query: (pageNumber) => `/listItems?page=${pageNumber}`,
      // Only have one cache entry because the arg always maps to one string
      serializeQueryArgs: ({ endpointName }) => {
        return endpointName
      },
      // Always merge incoming data to the cache entry
      merge: (currentCache, newItems) => {
        currentCache.push(...newItems)
      },
      // Refetch when the page arg changes
      forceRefetch({ currentArg, previousArg }) {
        return currentArg !== previousArg
      },
    }),
  }),
})

Source: RTK Documenation on the merge option

Using this you can easily implement infinite scroll. Changing the pageNumber parameter of your query, will automatically fetch new data and concat it with the data that was already in the cache.

To illustrate this, I've created a working example on CodeSandbox.

Answer 2

Here is a workaround for having infinite loading with caching benefits of rtk-query

in my example backend response type is

{ items: anyDTO[], count: number /* totalCount */ }

in order to make it work properly when invalidating tag I had to fetch first page with hook and handle the rest in useEffect.

import { useCallback, useEffect, useState } from 'react';
import { useDispatch } from 'react-redux';
import * as R from 'ramda';
import { ApiEndpointQuery } from '@reduxjs/toolkit/dist/query/core/module';
import { QueryHooks } from '@reduxjs/toolkit/dist/query/react/buildHooks';

interface UseLazeyInfiniteDataProps<T, N> {
  api: T;
  /** any rtk-query api: passing the whole enpoint so we have access to api utils to invalidate provided tags */
  apiEndpointName: N;
  /** apiEndpoint name to retrieve correct apiEndpoint query which will have 'initiate' and 'useQuery' */
  apiArgs: { [key: string]: any; params: object };
  /** apiArgs are the query arguments it should have a params objec */
  limit?: number;
  /** limit or page-size per request (defaults 20) */
  invalidatesTags?: any[];
}
/**
 * This hook is for having infinite loading experience with caching posibility of rtk-query
 * it's storing the data comming from rtk-q to local useState throgh a useEffect hook
 * in orther to make it work when invalidating tags it makes the first page request through rtk-query hook
 * and whenever it changes it will refetch the rest data
 */
const useLazyInfiniteData = <
  T extends { endpoints: any; util: any },
  N extends keyof T['endpoints'],
>({
  api,
  apiEndpointName,
  apiArgs,
  limit = 20,
  invalidatesTags,
}: UseLazeyInfiniteDataProps<T, N>) => {
  const dispatch = useDispatch<any>();
  const [pageNumber, setPageNumber] = useState(1); // first load only page 1
  const [maxPage, setMaxPage] = useState(0); // we don't know how many pages could exists yet
  const [accData, setAccData] = useState<any[]>([]);
  const [isFetchingMore, setIsFetchingMore] = useState(false);

  const apiEndpoint: ApiEndpointQuery<any, any> & QueryHooks<any> =
    api.endpoints[apiEndpointName];
  // we need this extra hook to automate refetching when invalidating tag
  // this will make the useEffect rerender if the first page data changes
  const {
    currentData: firstPageData,
    isLoading,
    isFetching,
    refetch: refetch_,
  } = apiEndpoint.useQuery({
    ...apiArgs,
    params: R.mergeRight(apiArgs.params, { offset: 0, limit }),
  });

  const refetch = useCallback(() => {
    if (invalidatesTags) {
      dispatch(api.util.invalidateTags());
    }
    refetch_();
  }, [api.util, dispatch, invalidatesTags, refetch_]);

  /** when params change like changing filters in the params then we reset the loading pages to 1 */
  useEffect(
    function resetPageLoadDataForSinglePage() {
      setPageNumber(1);
    },
    [apiArgs.params],
  );

  useEffect(
    function loadMoreDataOnPageNumberIncrease() {
      if (firstPageData)
        setMaxPage(Math.ceil((firstPageData as any).count / limit));

      if (pageNumber === 1) {
        setAccData((firstPageData as any)?.items ?? []);
      }
      if (pageNumber > 1) {
        setIsFetchingMore(true);
        const promises = R.range(1, pageNumber).map((page) =>
          dispatch(
            apiEndpoint.initiate({
              ...apiArgs,
              params: R.mergeRight(apiArgs.params, {
                offset: page * limit,
                limit,
              }),
            }),
          ).unwrap(),
        );

        Promise.all(promises)
          .then((data: any[]) => {
            const items = R.chain(R.propOr([], 'items'), [
              firstPageData,
              ...data,
            ]);
            setAccData(items);
          })
          .catch(console.error)
          .finally(() => {
            setIsFetchingMore(false);
          });
      }
    },
    [apiEndpoint, apiArgs, dispatch, firstPageData, limit, pageNumber],
  );

  /** increasing pageNumber will make the useEffect run */
  const loadMore = useCallback(() => {
    setPageNumber(R.inc);
  }, []);

  return {
    data: accData,
    loadMore,
    hasMore: pageNumber < maxPage,
    isLoading,
    isFetching,
    isFetchingMore,
    refetch,
  };
};

export default useLazyInfiniteData;

usage: Assuming you have rtk query API:

const extendedApi = emptySplitApi.injectEndpoints({ 
  endpoints: (build) => ({ 
    example: build.query({
      query: ({x, params: { offset, limit }}) => 'test'
    })
  }),
})

You can use it like:

useLazyInfiniteData({ 
  api: extendedApi,
  apiEndpointName: 'example',
  apiArgs: { x }, // better to be a memorized value
})