How to efficiently handle data with multiple filters in a shiny app

Question

We have created a shiny application where either the user can upload a big dataset (RData file over 200MB) or they can pick one from us. Then there are three different tabs where the user can filter the data (tab for numerics, tab for categorics)

So currently I have 3 reactive functions to serve that purpose. But downside is that the object is kept three times in memory. Is there a more efficient way to do this?

Please find a simplified example app below: note: in this app you only see 1 filter per tab. normally its more like this:

My_Filtered_Data[Species %in% input$filter1 &
                   x %in% input$x &
                   y %in% input$y &
                   z %in% input$z] #etc.

I was looking at reactiveValues but couldn't really find how it works. Reason I don't want to have it in 1 reactive is that everytime I change one of the filters on one of the sheets, the entire filtering process starts again and that is quite time consuming. I'd prefer to have one dataset that that gets updated with only the filter that is used at that time. That's the reason I included the different reactives

## app.R ##
library(shinydashboard)
library(data.table)
CustomHeader <- dashboardHeader(title='datatest')
iris<-iris

ui <- function(request) {
  dashboardPage(
    CustomHeader,
    ## Sidebar content
    dashboardSidebar(
      sidebarMenu(
        menuItem("filter1 & Import", tabName = "filter1", icon = icon("dashboard")),
        menuItem("filter2", tabName = "filter2", icon = icon("th")),
        menuItem("filter3", tabName = "filter3", icon = icon("th"))
      )
    ),
    ## Body content
    dashboardBody(
      tabItems(
        # First tab content
        tabItem(tabName = "filter1",
                fluidRow(box(width = 3,
                             selectInput(inputId = 'filter1','filter1:species',choices = unique(iris$Species))))

        ),

        tabItem(tabName = "filter2",
                fluidRow(box(width = 3,
                             sliderInput(inputId = 'filter2','filter2:Max.Sepal.Length',min = 0,max = 10,value = 10)
                                         ))

        ),
        tabItem(tabName = "filter3",
                fluidRow(box(width = 3,
                             sliderInput(inputId = 'filter3','filter3:Min.Sepal.Width',min = 0,max = 10,value = 0)
                             ),
                         box(width=9,dataTableOutput('mydata')))

        )
        )
      )
    )

}
server <- function(input, output) {
  My_Uploaded_Data <- reactive({
    My_Uploaded_Data<-data.table(iris)
    My_Uploaded_Data
  })

  My_Filtered_Data <- reactive({
    My_Filtered_Data<-My_Uploaded_Data()
    My_Filtered_Data[Species %in% input$filter1]
  })

  My_Filtered_Data2 <- reactive({
    My_Filtered_Data2<-My_Filtered_Data()
    My_Filtered_Data2[Sepal.Length < input$filter2]
  })  

  My_Filtered_Data3 <- reactive({
    My_Filtered_Data3<-My_Filtered_Data2()
    My_Filtered_Data3[Sepal.Width > input$filter3]
  })  
  output$mydata<-renderDataTable({
    My_Filtered_Data3()
  })

}
shinyApp(ui, server)

I was hoping something like tthis would work in reactiveValues

 react_vals <- reactiveValues(data = NULL)
  observe(react_vals$data <- MyLoadedData())
  observe(react_vals$data <- react_vals$data[Species %in% input$filter1])
  observe(react_vals$data <- react_vals$data[Sepal.Length < input$filter2])
  observe(react_vals$data <- react_vals$data[Sepal.Width > input$filter3])

EDIT: I also would like to include bookmarks: https://shiny.rstudio.com/articles/advanced-bookmarking.html and it seems you need reactiveValues for that. So another reason for me to move away from all these reactives/eventReactive

Answer 1

Instead of storing datasets in the reactive variables, just store the rows which qualify. That way, each reactive value is only replaced when it's filter changes; they aren't linked together. The output just uses the rows which pass all filters.

At the top of the program, change iris to a data.table:

library(shinydashboard)
library(data.table)
CustomHeader <- dashboardHeader(title = 'datatest')
iris <- iris
setDT(iris)  # Added

Then use this for the server logic:

server <- function(input, output) {
  filter1_rows <- reactive({
    iris[Species %in% input$filter1,   which = TRUE]
  })
  filter2_rows <- reactive({
    iris[Sepal.Length < input$filter2, which = TRUE]
  })
  filter3_rows <- reactive({
    iris[Sepal.Width > input$filter3,  which = TRUE]
  })
  output$mydata <- renderDataTable({
    final_rows <- intersect(filter1_rows(), filter2_rows())
    final_rows <- intersect(final_rows,     filter3_rows())
    iris[final_rows]
  })
}

This uses the often-overlooked which argument for data.table[...], which means only the row numbers of the subsetted table should be returned.

Answer 2

I think your problem has nothing to do with shiny and/or reactive programming. It's a "classic time vs memory" situation. Basically speaking you have only two options: Store "partially" filtered objects or not.

If you do store them, you use a lot of memory but can return the object instantly. If not, you need only store the original object but you have to filter it everytime again. There is nothing in between. You just cannot create an object that is different from the original (i.e. filtered) but takes no additional memory, not even with reactiveValues.

Of course you can do tradeoffs, e.g. creating an intermediate object for the first filter and computing the second and the third filter on-the-fly, but that does not change the underlying problem.