Android Paging 3 how to filter, sort and search my data

Question

I'm trying to implement paging I'm using Room and it took me ages to realize that its all done for me 😆 but what I need to do is be able to filter search and sort my data. I want to keep it as LiveData for now I can swap to flow later. I had this method to filter search and sort and it worked perfectly,

    @SuppressLint("DefaultLocale")
    private fun searchAndFilterPokemon(): LiveData<List<PokemonWithTypesAndSpecies>> {
        return Transformations.switchMap(search) { search ->
            val allPokemon = repository.searchPokemonWithTypesAndSpecies(search)
            Transformations.switchMap(filters) { filters ->
                val pokemon = when {
                    filters.isNullOrEmpty() -> allPokemon
                    else -> {
                        Transformations.switchMap(allPokemon) { pokemonList ->
                            val filteredList = pokemonList.filter { pokemon ->
                                pokemon.matches = 0
                                val filter = filterTypes(pokemon, filters)
                                filter
                            }
                            maybeSortList(filters, filteredList)
                        }
                    }
                }
                pokemon
            }
        }
    }

It have a few switchmaps here, the first is responding to search updating

    var search: MutableLiveData<String> = getSearchState()

the second is responding to filters updating

    val filters: MutableLiveData<MutableSet<String>> = getCurrentFiltersState()

and the third is watching the searched list updating, it then calls filterTypes and maybeSortList which are small methods for filtering and sorting

    @SuppressLint("DefaultLocale")
    private fun filterTypes(
        pokemon: PokemonWithTypesAndSpecies,
        filters: MutableSet<String>
    ): Boolean {
        var match = false
        for (filter in filters) {
            for (type in pokemon.types) {
                if (type.name.toLowerCase() == filter.toLowerCase()) {
                    val matches = pokemon.matches.plus(1)
                    pokemon.apply {
                        this.matches = matches
                    }
                    match = true
                }
            }
        }
        return match
    }

    private fun maybeSortList(
        filters: MutableSet<String>,
        filteredList: List<PokemonWithTypesAndSpecies>
    ): MutableLiveData<List<PokemonWithTypesAndSpecies>> {
        return if (filters.size > 1)
            MutableLiveData(filteredList.sortedByDescending {
                Log.d("VM", "SORTING ${it.pokemon.name} ${it.matches}")
                it.matches
            })
        else MutableLiveData(filteredList)
    }

as mentioned I want to migrate these to paging 3 and am having difficulty doing it Ive changed my repository and dao to return a PagingSource and I just want to change my view model to return the PagingData as a live data, so far I have this

@SuppressLint("DefaultLocale")
private fun searchAndFilterPokemonPager(): LiveData<PagingData<PokemonWithTypesAndSpecies>> {
    val pager =  Pager(
        config = PagingConfig(
            pageSize = 50,
            enablePlaceholders = false,
            maxSize = 200
        )
    ){
        searchAndFilterPokemonWithPaging()
    }.liveData.cachedIn(viewModelScope)

    Transformations.switchMap(filters){
        MutableLiveData<String>()
    }

    return Transformations.switchMap(search) {search ->
        val searchedPokemon =
            MutableLiveData<PagingData<PokemonWithTypesAndSpecies>>(pager.value?.filter { it.pokemon.name.contains(search) })
        Transformations.switchMap(filters) { filters ->
            val pokemon = when {
                filters.isNullOrEmpty() -> searchedPokemon
                else -> {
                    Transformations.switchMap(searchedPokemon) { pokemonList ->
                        val filteredList = pokemonList.filter { pokemon ->
                            pokemon.matches = 0
                            val filter = filterTypes(pokemon, filters)
                            filter
                        }
                        maybeSortList(filters, filteredList = filteredList)
                    }
                }
            }
            pokemon
        }
    }
}

but the switchmap is giving me an error that

Type inference failed: Cannot infer type parameter Y in 

fun <X : Any!, Y : Any!> switchMap
(
    source: LiveData<X!>,
    switchMapFunction: (input: X!) → LiveData<Y!>!
 )

which I think I understand but am not sure how to fix it, also the filter and sort methods won't work anymore and I cant see any good method replacements for it with the PageData, it has a filter but not a sort? any help appreciated : LiveData<Y!

UPDATE thanks to @Shadow I've rewritten it to implement searching using a mediator live data but im still stuck on filtering

    init {
        val combinedValues =
            MediatorLiveData<Pair<String?, MutableSet<String>?>?>().apply {
                addSource(search) {
                    value = Pair(it, filters.value)
                }
                addSource(filters) {
                    value = Pair(search.value, it)
                }
            }

        searchPokemon = Transformations.switchMap(combinedValues) { pair ->
            val search = pair?.first
            val filters = pair?.second
            if (search != null && filters != null) {
                searchAndFilterPokemonPager(search)
            } else null
        }
    }



    @SuppressLint("DefaultLocale")
    private fun searchAndFilterPokemonPager(search: String): LiveData<PagingData<PokemonWithTypesAndSpecies>> {
        return Pager(
            config = PagingConfig(
                pageSize = 50,
                enablePlaceholders = false,
                maxSize = 200
            )
        ){
            searchAllPokemonWithPaging(search)
        }.liveData.cachedIn(viewModelScope)

    }

    @SuppressLint("DefaultLocale")
    private fun searchAllPokemonWithPaging(search: String): PagingSource<Int, PokemonWithTypesAndSpecies> {
        return repository.searchPokemonWithTypesAndSpeciesWithPaging(search)
    }

Answer 1

I do not believe you can sort correctly on the receiving end when you are using a paging source. This is because paging will return a chunk of data from the database in whichever order it is, or in whichever order you specify in its query directly.

Say you have a list of names in the database and you want to display them sorted alphabetically.

Unless you actually specify that sorting in the query itself, the default query will fetch the X first names (X being the paging size you have configured) it finds in whichever order the database is using for the results of that query.

You can sort the results alright, but it will only sort those X first names it returned. That means if you had any names that started with A and they happened to not come in that chunk of names, they will only show when you have scrolled far enough for them to be loaded by the pager, and only then they will show sorted correctly in the present list. You might see names moving around as a result of this whenever a new page is loaded into your list.

That's for sorting, now for search.

What I ended up doing for my search was to just throw away the database's own capability for searching. You can use " LIKE " in queries directly, but unless your search structure is very basic it will be useless. There is also Fts4 available: https://developer.android.com/reference/androidx/room/Fts4

But it is such a PIA to setup and make use of that I ended up seeing absolutely no reward worth the effort for my case.

So I just do the search however I want on the receiving end instead using a Transformations.switchMap to trigger a new data fetch from the database whenever the user input changes coupled with the filtering on the data I receive.

You already have part of this implemented, just take out the contents of the Transformations.switchMap(filters) and simply return the data, then you conduct the search on the results returned in the observer that is attached to the searchAndFilterPokemonPager call.

Filter is the same logic as search too, but I would suggest to make sure to filter first before searching since typically search is input driven and if you don't add a debouncer it will be triggering a new search for every character the user enters or deletes.

In short:

1. sort in the query directly so the results you receive are already sorted
2. implement a switchMap attached to the filter value to trigger a new data fetch with the new filter value taken into account
3. implement a switchMap just like the filter, but for the search input
4. filter the returned data right before you submit to your list/recyclerview adapter
5. same as above, but for search