Efficiently working with large data sets in Vue applications with Vuex

Question

In my Vue application, I have Vuex store modules with large arrays of resource objects in their state. To easily access individual resources in those arrays, I make Vuex getter functions that map resources or lists of resources to various keys (e.g. 'id' or 'tags'). This leads to sluggish performance and a huge memory memory footprint. How do I get the same functionality and reactivity without so much duplicated data?

Store Module Example

export default {
  state: () => ({
    all: [
      { id: 1, tags: ['tag1', 'tag2'] },
      ...
    ],
    ...
  }),

  ...

  getters: {
    byId: (state) => {
      return state.all.reduce((map, item) => {
        map[item.id] = item
        return map
      }, {})
    },

    byTag: (state) => {
      return state.all.reduce((map, item, index) => {
        for (let i = 0; i < item.tags.length; i++) {
          map[item.tags[i]] = map[item.tags[i]] || []
          map[item.tags[i]].push(item)
        }
        return map
      }, {})
    },
  }
}

Component Example

export default {
  ...,

  data () {
    return {
      itemId: 1
    }
  },

  computed: {
    item () {
      return this.$store.getters['path/to/byId'][this.itemId]
    },

    relatedItems () {
      return this.item && this.item.tags.length
        ? this.$store.getters['path/to/byTag'][this.item.tags[0]]
        : []
    }
  }
}

Answer 1

To fix this problem, look to an old, standard practice in programming: indexing. Instead of storing a map with the full item values duplicated in the getter, you can store a map to the index of the item in state.all. Then, you can create a new getter that returns a function to access a single item. In my experience, the indexing getter functions always run faster than the old getter functions, and their output takes up a lot less space in memory (on average 80% less in my app).

New Store Module Example

export default {
  state: () => ({
    all: [
      { id: 1, tags: ['tag1', 'tag2'] },
      ...
    ],
    ...
  }),

  ...

  getters: {
    indexById: (state) => {
      return state.all.reduce((map, item, index) => {
        // Store the `index` instead of the `item`
        map[item.id] = index
        return map
      }, {})
    },

    byId: (state, getters) => (id) => {
      return state.all[getters.indexById[id]]
    },

    indexByTags: (state) => {
      return state.all.reduce((map, item, index) => {
        for (let i = 0; i < item.tags.length; i++) {
          map[item.tags[i]] = map[item.tags[i]] || []
          // Again, store the `index` not the `item`
          map[item.tags[i]].push(index)
        }
        return map
      }, {})
    },

    byTag: (state, getters) => (tag) => {
      return (getters.indexByTags[tag] || []).map(index => state.all[index])
    }
  }
}

New Component Example

export default {
  ...,

  data () {
    return {
      itemId: 1
    }
  },

  computed: {
    item () {
      return this.$store.getters['path/to/byId'](this.itemId)
    },

    relatedItems () {
      return this.item && this.item.tags.length
        ? this.$store.getters['path/to/byTag'](this.item.tags[0])
        : []
    }
  }
}

The change seems small, but it makes a huge difference in terms of performance and memory efficiency. It is still fully reactive, just as before, but you're no longer duplicating all of the resource objects in memory. In my implementation, I abstracted out the various indexing methodologies and index expansion methodologies to make the code very maintainable.

You can check out a full proof of concept on github, here: https://github.com/aidangarza/vuex-indexed-getters

Answer 2

While only storing select fields is a good intermediate option (per @aidangarza), it's still not viable when you end up with really huge sets of data. E.g. actively working with 2 million records of "just 2 fields" will still eat your memory and ruin browser performance.

In general, when working with large (or unpredictable) data sets in Vue (using VueX), simply skip the get and commit mechanisms altogether. Keep using VueX to centralize your CRUD operations (via Actions), but do not try to "cache" the results, rather let each component cache what they need for as long as they're using it (e.g. the current working page of the results, some projection thereof, etc.).

In my experience VueX caching is intended for reasonably bounded data, or bounded subsets of data in the current usage context (i.e. for a currently logged in user). When you have data where you have no idea about its scale, then keep its access on an "as needed" basis by your Vue components via Actions only; no getters or mutations for those potentially huge data sets.