Jetpack Paging3 library makes several calls to the same page

Question

I am trying to implement Jetpack paging 3 library following the codelab using room database as source of truth and a RemoteMediator. The app queries the google books api but for some reason when I perform a search it makes several calls to the same page. For example I get this in the log when I search fire without scrolling:

D/BooksRepository: new search: fire
D/BooksRemoteMediator: title: fire, page: 0
I/okhttp.OkHttpClient: --> GET https://www.googleapis.com/books/v1/volumes?q=intitle%3Afire&key=AIzaSyBxHmT9nFCp9n2uOHkS3Gcq2OO3zbxaMrw&maxResults=40&startIndex=0
I/okhttp.OkHttpClient: <-- 200 https://www.googleapis.com/books/v1/volumes?q=intitle%3Afire&key=AIzaSyBxHmT9nFCp9n2uOHkS3Gcq2OO3zbxaMrw&maxResults=40&startIndex=0 (648ms, unknown-length body)
D/BooksRemoteMediator: title: fire, page: 1
I/okhttp.OkHttpClient: --> GET https://www.googleapis.com/books/v1/volumes?q=intitle%3Afire&key=AIzaSyBxHmT9nFCp9n2uOHkS3Gcq2OO3zbxaMrw&maxResults=40&startIndex=1
I/okhttp.OkHttpClient: <-- 200 https://www.googleapis.com/books/v1/volumes?q=intitle%3Afire&key=AIzaSyBxHmT9nFCp9n2uOHkS3Gcq2OO3zbxaMrw&maxResults=40&startIndex=1 (608ms, unknown-length body)
D/BooksRemoteMediator: title: fire, page: 0
I/okhttp.OkHttpClient: --> GET https://www.googleapis.com/books/v1/volumes?q=intitle%3Afire&key=AIzaSyBxHmT9nFCp9n2uOHkS3Gcq2OO3zbxaMrw&maxResults=40&startIndex=0
I/okhttp.OkHttpClient: <-- 200 https://www.googleapis.com/books/v1/volumes?q=intitle%3Afire&key=AIzaSyBxHmT9nFCp9n2uOHkS3Gcq2OO3zbxaMrw&maxResults=40&startIndex=0 (629ms, unknown-length body)
D/BooksRemoteMediator: title: fire, page: 1
I/okhttp.OkHttpClient: --> GET https://www.googleapis.com/books/v1/volumes?q=intitle%3Afire&key=AIzaSyBxHmT9nFCp9n2uOHkS3Gcq2OO3zbxaMrw&maxResults=40&startIndex=1
I/okhttp.OkHttpClient: <-- 200 https://www.googleapis.com/books/v1/volumes?q=intitle%3Afire&key=AIzaSyBxHmT9nFCp9n2uOHkS3Gcq2OO3zbxaMrw&maxResults=40&startIndex=1 (843ms, unknown-length body)
D/BooksRemoteMediator: title: fire, page: 0
I/okhttp.OkHttpClient: --> GET https://www.googleapis.com/books/v1/volumes?q=intitle%3Afire&key=AIzaSyBxHmT9nFCp9n2uOHkS3Gcq2OO3zbxaMrw&maxResults=40&startIndex=0
I/okhttp.OkHttpClient: <-- 200 https://www.googleapis.com/books/v1/volumes?q=intitle%3Afire&key=AIzaSyBxHmT9nFCp9n2uOHkS3Gcq2OO3zbxaMrw&maxResults=40&startIndex=0 (527ms, unknown-length body)
D/BooksRemoteMediator: title: fire, page: 1
I/okhttp.OkHttpClient: --> GET https://www.googleapis.com/books/v1/volumes?q=intitle%3Afire&key=AIzaSyBxHmT9nFCp9n2uOHkS3Gcq2OO3zbxaMrw&maxResults=40&startIndex=1
I/okhttp.OkHttpClient: <-- 200 https://www.googleapis.com/books/v1/volumes?q=intitle%3Afire&key=AIzaSyBxHmT9nFCp9n2uOHkS3Gcq2OO3zbxaMrw&maxResults=40&startIndex=1 (734ms, unknown-length body)
D/BooksRemoteMediator: title: fire, page: 2
I/okhttp.OkHttpClient: --> GET https://www.googleapis.com/books/v1/volumes?q=intitle%3Afire&key=AIzaSyBxHmT9nFCp9n2uOHkS3Gcq2OO3zbxaMrw&maxResults=40&startIndex=2
I/okhttp.OkHttpClient: <-- 200 https://www.googleapis.com/books/v1/volumes?q=intitle%3Afire&key=AIzaSyBxHmT9nFCp9n2uOHkS3Gcq2OO3zbxaMrw&maxResults=40&startIndex=2 (783ms, unknown-length body)
D/BooksRemoteMediator: title: fire, page: 3
I/okhttp.OkHttpClient: --> GET https://www.googleapis.com/books/v1/volumes?q=intitle%3Afire&key=AIzaSyBxHmT9nFCp9n2uOHkS3Gcq2OO3zbxaMrw&maxResults=40&startIndex=3
I/okhttp.OkHttpClient: <-- 200 https://www.googleapis.com/books/v1/volumes?q=intitle%3Afire&key=AIzaSyBxHmT9nFCp9n2uOHkS3Gcq2OO3zbxaMrw&maxResults=40&startIndex=3 (769ms, unknown-length body)
D/BooksRemoteMediator: title: fire, page: 2
I/okhttp.OkHttpClient: --> GET https://www.googleapis.com/books/v1/volumes?q=intitle%3Afire&key=AIzaSyBxHmT9nFCp9n2uOHkS3Gcq2OO3zbxaMrw&maxResults=40&startIndex=2
I/okhttp.OkHttpClient: <-- 200 https://www.googleapis.com/books/v1/volumes?q=intitle%3Afire&key=AIzaSyBxHmT9nFCp9n2uOHkS3Gcq2OO3zbxaMrw&maxResults=40&startIndex=2 (521ms, unknown-length body)
D/BooksRemoteMediator: title: fire, page: 3
I/okhttp.OkHttpClient: --> GET https://www.googleapis.com/books/v1/volumes?q=intitle%3Afire&key=AIzaSyBxHmT9nFCp9n2uOHkS3Gcq2OO3zbxaMrw&maxResults=40&startIndex=3
I/okhttp.OkHttpClient: <-- 200 https://www.googleapis.com/books/v1/volumes?q=intitle%3Afire&key=AIzaSyBxHmT9nFCp9n2uOHkS3Gcq2OO3zbxaMrw&maxResults=40&startIndex=3 (549ms, unknown-length body)
D/BooksRemoteMediator: title: fire, page: 2
I/okhttp.OkHttpClient: --> GET https://www.googleapis.com/books/v1/volumes?q=intitle%3Afire&key=AIzaSyBxHmT9nFCp9n2uOHkS3Gcq2OO3zbxaMrw&maxResults=40&startIndex=2
I/okhttp.OkHttpClient: <-- 200 https://www.googleapis.com/books/v1/volumes?q=intitle%3Afire&key=AIzaSyBxHmT9nFCp9n2uOHkS3Gcq2OO3zbxaMrw&maxResults=40&startIndex=2 (966ms, unknown-length body)
D/BooksRemoteMediator: title: fire, page: 3
I/okhttp.OkHttpClient: --> GET https://www.googleapis.com/books/v1/volumes?q=intitle%3Afire&key=AIzaSyBxHmT9nFCp9n2uOHkS3Gcq2OO3zbxaMrw&maxResults=40&startIndex=3
I/okhttp.OkHttpClient: <-- 200 https://www.googleapis.com/books/v1/volumes?q=intitle%3Afire&key=AIzaSyBxHmT9nFCp9n2uOHkS3Gcq2OO3zbxaMrw&maxResults=40&startIndex=3 (673ms, unknown-length body)
D/BooksRemoteMediator: title: fire, page: 4
I/okhttp.OkHttpClient: --> GET https://www.googleapis.com/books/v1/volumes?q=intitle%3Afire&key=AIzaSyBxHmT9nFCp9n2uOHkS3Gcq2OO3zbxaMrw&maxResults=40&startIndex=4
I/okhttp.OkHttpClient: <-- 200 https://www.googleapis.com/books/v1/volumes?q=intitle%3Afire&key=AIzaSyBxHmT9nFCp9n2uOHkS3Gcq2OO3zbxaMrw&maxResults=40&startIndex=4 (634ms, unknown-length body)
D/BooksRemoteMediator: title: fire, page: 5
I/okhttp.OkHttpClient: --> GET https://www.googleapis.com/books/v1/volumes?q=intitle%3Afire&key=AIzaSyBxHmT9nFCp9n2uOHkS3Gcq2OO3zbxaMrw&maxResults=40&startIndex=5
I/okhttp.OkHttpClient: <-- 200 https://www.googleapis.com/books/v1/volumes?q=intitle%3Afire&key=AIzaSyBxHmT9nFCp9n2uOHkS3Gcq2OO3zbxaMrw&maxResults=40&startIndex=5 (604ms, unknown-length body)
D/BooksRemoteMediator: title: fire, page: 4
I/okhttp.OkHttpClient: --> GET https://www.googleapis.com/books/v1/volumes?q=intitle%3Afire&key=AIzaSyBxHmT9nFCp9n2uOHkS3Gcq2OO3zbxaMrw&maxResults=40&startIndex=4
I/okhttp.OkHttpClient: <-- 200 https://www.googleapis.com/books/v1/volumes?q=intitle%3Afire&key=AIzaSyBxHmT9nFCp9n2uOHkS3Gcq2OO3zbxaMrw&maxResults=40&startIndex=4 (632ms, unknown-length body)
D/BooksRemoteMediator: title: fire, page: 3
I/okhttp.OkHttpClient: --> GET https://www.googleapis.com/books/v1/volumes?q=intitle%3Afire&key=AIzaSyBxHmT9nFCp9n2uOHkS3Gcq2OO3zbxaMrw&maxResults=40&startIndex=3
I/okhttp.OkHttpClient: <-- 200 https://www.googleapis.com/books/v1/volumes?q=intitle%3Afire&key=AIzaSyBxHmT9nFCp9n2uOHkS3Gcq2OO3zbxaMrw&maxResults=40&startIndex=3 (602ms, unknown-length body)

My implementation is like this:

Repository:

class BooksRepository(private val service: BookService, private val database: BooksDatabase) {
    companion object {
        private const val NETWORK_PAGE_SIZE = 40
    }

    fun getSearchResultStream(
        title: String = "",
        author: String = "",
        publisher: String = "",
        isbn: String = ""
    ): Flow<PagingData<Book>> {
        Timber.d("new search: $title")
        val dbQuery = "%${title.replace(' ', '%')}%"
        val pagingSourceFactory =  { database.bookDao.getBooks(dbQuery, author, publisher)}

        return Pager(
            config = PagingConfig(
                pageSize = NETWORK_PAGE_SIZE,
                enablePlaceholders = false
            ),
            remoteMediator = BooksRemoteMediator(title, author, publisher, isbn, apiKey, service, database),
            pagingSourceFactory = pagingSourceFactory
        ).flow
    }
}

RemoteMediator:

private const val BOOKS_STARTING_PAGE_INDEX = 0
@OptIn(ExperimentalPagingApi::class)
class BooksRemoteMediator(
    private val title: String?,
    private val author: String?,
    private val publisher: String?,
    private val isbn: String?,
    private val key: String,
    private val service: BookService,
    private val booksDatabase: BooksDatabase
) : RemoteMediator<Int, Book>() {

    override suspend fun load(loadType: LoadType, state: PagingState<Int, Book>): MediatorResult {
        val page = when (loadType) {
            LoadType.REFRESH -> {
                val remoteKeys = getRemoteKeyClosestToCurrentPosition(state)
                remoteKeys?.nextKey?.minus(1) ?: BOOKS_STARTING_PAGE_INDEX
            }
            LoadType.PREPEND -> {
                val remoteKeys = getRemoteKeyForFirstItem(state)
                if (remoteKeys == null) {
                    // The LoadType is PREPEND so some data was loaded before,
                    // so we should have been able to get remote keys
                    // If the remoteKeys are null, then we're an invalid state and we have a bug
                    throw InvalidObjectException("Remote key and the prevKey should not be null")
                }
                // If the previous key is null, then we can't request more data
                val prevKey = remoteKeys.prevKey
                if (prevKey == null) {
                    return MediatorResult.Success(endOfPaginationReached = true)
                }
                remoteKeys.prevKey
            }
            LoadType.APPEND -> {
                val remoteKeys = getRemoteKeyForLastItem(state)
                if (remoteKeys == null || remoteKeys.nextKey == null) {
                    throw InvalidObjectException("Remote key should not be null for $loadType")
                }
                remoteKeys.nextKey
            }
        }

        Timber.d("title: $title, page: $page")
        val sb = StringBuilder()
        if (!title.isNullOrBlank()) sb.append("$TITLE$title+")
        if (!author.isNullOrBlank()) sb.append("$AUTHOR$author+")
        if (!publisher.isNullOrBlank()) sb.append("$PUBLISHER$publisher+")
        if (!isbn.isNullOrBlank()) sb.append("$ISBN$isbn+")
        sb.setLength(sb.length - 1)
        val apiQuery = sb.toString()

        try {
            val apiResponse = service.searchBooks(apiQuery, key, state.config.pageSize, page)

            val books = apiResponse.items
            val endOfPaginationReached = books.isEmpty()
            booksDatabase.withTransaction {
                // clear all tables in the database
                if (loadType == LoadType.REFRESH) {
                    booksDatabase.remoteKeysDao.clearRemoteKeys()
                    booksDatabase.bookDao.clearBooks()
                }
                val prevKey = if (page == BOOKS_STARTING_PAGE_INDEX) null else page - 1
                val nextKey = if (endOfPaginationReached) null else page + 1
                val keys = books.map {
                    RemoteKeys(bookId = it.id, prevKey = prevKey, nextKey = nextKey)
                }
                booksDatabase.remoteKeysDao.insertAll(keys)
                booksDatabase.bookDao.insert(books)
            }
            return MediatorResult.Success(endOfPaginationReached = endOfPaginationReached)
        } catch (exception: IOException) {
            return MediatorResult.Error(exception)
        } catch (exception: HttpException) {
            return MediatorResult.Error(exception)
        }
    }

    private suspend fun getRemoteKeyForLastItem(state: PagingState<Int, Book>): RemoteKeys? {
        // Get the last page that was retrieved, that contained items.
        // From that last page, get the last item
        return state.pages.lastOrNull() { it.data.isNotEmpty() }?.data?.lastOrNull()
            ?.let { book ->
                // Get the remote keys of the last item retrieved
                booksDatabase.remoteKeysDao.remoteKeysBookId(book.id)
            }
    }

    private suspend fun getRemoteKeyForFirstItem(state: PagingState<Int, Book>): RemoteKeys? {
        // Get the first page that was retrieved, that contained items.
        // From that first page, get the first item
        return state.pages.firstOrNull { it.data.isNotEmpty() }?.data?.firstOrNull()
            ?.let { book ->
                // Get the remote keys of the first items retrieved
                booksDatabase.remoteKeysDao.remoteKeysBookId(book.id)
            }
    }

    private suspend fun getRemoteKeyClosestToCurrentPosition(
        state: PagingState<Int, Book>
    ): RemoteKeys? {
        // The paging library is trying to load data after the anchor position
        // Get the item closest to the anchor position
        return state.anchorPosition?.let { position ->
            state.closestItemToPosition(position)?.id?.let { bookId ->
                booksDatabase.remoteKeysDao.remoteKeysBookId(bookId)
            }
        }
    }

}

ViewModel:

class BookListViewModel(private val repository: BooksRepository) : ViewModel() {
    private var currentQueryValue: String? = null

    private var currentSearchResult: Flow<PagingData<Book>>? = null

    fun searchRepo(queryString: String): Flow<PagingData<Book>> {
        val lastResult = currentSearchResult
        if (queryString == currentQueryValue && lastResult != null) {
            return lastResult
        }
        currentQueryValue = queryString
        val newResult: Flow<PagingData<Book>> = repository.getSearchResultStream(queryString)
            .cachedIn(viewModelScope)
        currentSearchResult = newResult
        return newResult
    }
}

Fragment:

class BookListFragment : Fragment() {
...

    override fun onCreateView(inflater: LayoutInflater, container: ViewGroup?, savedInstanceState: Bundle?): View? {
        setHasOptionsMenu(true)
        binding = FragmentBookListBinding.inflate(inflater, container, false).apply {
            lifecycleOwner = viewLifecycleOwner
            viewModel = bookListViewModel
        }
        initAdapter()

        val queryString = queryArgs.split(",")
        val query = savedInstanceState?.getString(LAST_SEARCH_QUERY) ?: DEFAULT_QUERY
        search(query)

        setHasOptionsMenu(true)
        return binding.root
    }

    override fun onActivityCreated(savedInstanceState: Bundle?) {
        super.onActivityCreated(savedInstanceState)
        binding.retryButton.setOnClickListener { adapter.retry() }
    }

    override fun onSaveInstanceState(outState: Bundle) {
        super.onSaveInstanceState(outState)
        outState.putString(LAST_SEARCH_QUERY, latest)
    }

    private fun search(query: String) {
        // Make sure we cancel the previous job before creating a new one
        searchJob?.cancel()
        searchJob = lifecycleScope.launch {
            bookListViewModel.searchRepo(query).collectLatest{
                adapter.submitData(it)
            }
        }
    }

    private fun initAdapter() {
        binding.rvBooks.adapter = adapter.withLoadStateHeaderAndFooter(
            header = BooksLoadStateAdapter { adapter.retry() },
            footer = BooksLoadStateAdapter { adapter.retry() }
        )

        adapter.addLoadStateListener { loadState ->
            // Only show the list if refresh succeeds.
            binding.rvBooks.isVisible = loadState.source.refresh is LoadState.NotLoading
            // Show loading spinner during initial load or refresh.
            binding.pbLoading.isVisible = loadState.source.refresh is LoadState.Loading
            // Show the retry state if initial load or refresh fails.
            binding.retryButton.isVisible = loadState.source.refresh is LoadState.Error

            // Toast on any error, regardless of whether it came from RemoteMediator or PagingSource
            val errorState = loadState.source.append as? LoadState.Error
                ?: loadState.source.prepend as? LoadState.Error
                ?: loadState.append as? LoadState.Error
                ?: loadState.prepend as? LoadState.Error
            errorState?.let {
                Toast.makeText(
                    context,
                    "\uD83D\uDE28 Wooops ${it.error}",
                    Toast.LENGTH_LONG
                ).show()
            }
        }
    }

    private fun initSearch(menu: Menu) {
        val searchItem = menu.findItem(R.id.action_search)
        val searchView = searchItem.actionView as android.widget.SearchView
        searchView.setOnQueryTextListener(object : SearchView.OnQueryTextListener, android.widget.SearchView.OnQueryTextListener {
            override fun onQueryTextSubmit(query: String?): Boolean {
                try {
                    latest = query!!
                    updateBookListFromInput(query)
                } catch (e: Exception) {
                    Timber.d(e)
                }
                return false
            }

            override fun onQueryTextChange(newText: String?): Boolean {
                return false
            }
        })
        lifecycleScope.launch {
            adapter.loadStateFlow
                // Only emit when REFRESH LoadState changes.
                .distinctUntilChangedBy { it.refresh }
                // Only react to cases where REFRESH completes i.e., NotLoading.
                .filter { it.refresh is LoadState.NotLoading }
                .collect { binding.rvBooks.scrollToPosition(0) }
        }
    }

    private fun updateBookListFromInput(query: String?) {
        query?.trim().let {
            if (!it.isNullOrEmpty()) {
                search(it.toString())
            }
        }
    }


    override fun onCreateOptionsMenu(menu: Menu, inflater: MenuInflater) {
        inflater.inflate(R.menu.book_list_menu, menu)
        initSearch(menu)
        _menu = menu
        val recentList: ArrayList<String> = SpUtil.getQueryList(requireContext())
        var recentMenu: MenuItem? = null
        for (item in recentList) {
            recentMenu = menu.add(Menu.NONE, recentList.indexOf(item), Menu.NONE, item)
        }
    }

    companion object {
        private const val LAST_SEARCH_QUERY: String = "last_search_query"
        private const val DEFAULT_QUERY = "Fishing"
    }
}

Dao:

@Dao
interface BooksDao {
    @Insert(onConflict = OnConflictStrategy.REPLACE)
    suspend fun insert(books: List<Book>)

    @Query("DELETE FROM books")
    suspend fun clearBooks()

    @Query("SELECT * FROM books WHERE (title LIKE :title) OR (authors LIKE :author) " +
            "OR (publisher LIKE :publisher) ORDER BY title ASC")
    fun getBooks(title: String, author: String = "", publisher: String = ""): PagingSource<Int, Book>
}

Service:

interface BookService {
    @GET("volumes")
    suspend fun searchBooks(
        @Query("q") query: String,
        @Query("key") apiKey: String,
        @Query("maxResults") max: Int,
        @Query("startIndex") page: Int
    ): BookSearchResponse
}

It would be great if someone point out what I am doing wrong and help fix this problem. Thank you

Answer 1

I spent some time with this problem and after reading user9694585 and dunkypie answers, I was able to solve my problem. In my case, I was using the movieDb API, exactly as Doilio Matsinhe, and I discover that I can't rely on the API id for two reasons:

• The remote ids were out of order on the same page. So if I save them on the table as is, I lose the order from the server.
• If the ids are out of order I couldn't reach the real first/last items of the page. I think this part that makes the mediator be in a loop.

So what I ended up doing was making the remote id as an attribute and creating a primary key with autoGenerate = true on movie table:

@Entity(tableName = "movie")    
data class MovieTable(
    @PrimaryKey(autoGenerate = true)
    var id: Long = 0,
    var remoteId: Long = 0,
    ...
)

@Dao
interface RemoteKeysDao {
   ...
    @Query("SELECT * FROM remote_keys WHERE movieId = :movieId")
    suspend fun remoteKeyId(movieId: Long): RemoteKeyTable?
   ...
}

My RemoteMediator is very similar to Doilio Matsinhe. So I leave only the methods that were not in his answer:

private suspend fun getRemoteKeyForLastItem(state: PagingState<Int, MovieTable>): RemoteKeyTable? {
    return state.pages
        .lastOrNull {
            it.data.isNotEmpty()
        }?.data?.lastOrNull()
        ?.let { movie ->
            remoteKeyLocalDataSource.remoteKeyId(movie.remoteId)
        }
}

private suspend fun getRemoteKeyForFirstItem(state: PagingState<Int, MovieTable>): RemoteKeyTable? {
    return state.pages
        .firstOrNull {
            it.data.isNotEmpty()
        }?.data?.firstOrNull()
        ?.let { movie ->
            remoteKeyLocalDataSource.remoteKeyId(movie.remoteId)
        }
}

private suspend fun getRemoteKeyClosestToCurrentPosition(
    state: PagingState<Int, MovieTable>
): RemoteKeyTable? {
    return state.anchorPosition?.let { position ->
        state.closestItemToPosition(position)?.remoteId?.let { id ->
            remoteKeyLocalDataSource.remoteKeyId(id)
        }
    }
}

Basically is the one that we have on the codelab but using the movie.remoteId to get the remoteKey.id.