PyQt4 force view to fetchMore from QAbstractItemModel

Question

I have a QTableView that dynamically loads data from a custom model that inherits QAbstractItemModel. The model implements both fetchMore and canFetchMore.

The problem is that I would like to be able to select all rows for small datasets, but if I hit ctrl-a in the view it only will select the rows that are currently loaded.

Is there some mechanism to force the QTableView to fetch more rows? Ideally I would like to show a progress bar indicating the fraction of data that has been loaded from the model. Every few seconds I would like to force the model to load a bit more of the data, but I still want to let the user interact with the data that has been loaded so far. This way when the progress bar is complete the user can press ctrl-a and be confident that all data is selected.

---

Edit: I have another motivating use case. I want to jump to a specific row, but if that row is not loaded my interface does nothing.

How can I force a QAbstractItemModel to fetch more (or up to a specific row) and then force the QTableView to show it?

If I don't implement fetchMore and canFetchMore, the previous functionality works, but loading the tables is very slow. When I implement those methods the opposite happens. Not having an answer to this problem is causing issues with the usability of my qt interface, so I'm opening a bounty for this question.

Here is a method I'm using to select a specific row.

def select_row_from_id(view, _id, scroll=False, collapse=True):
    """
        _id is from the iders function (i.e. an ibeis rowid)
        selects the row in that view if it exists
    """
    with ut.Timer('[api_item_view] select_row_from_id(id=%r, scroll=%r, collapse=%r)' %
                  (_id, scroll, collapse)):
        qtindex, row = view.get_row_and_qtindex_from_id(_id)
        if row is not None:
            if isinstance(view, QtWidgets.QTreeView):
                if collapse:
                    view.collapseAll()
                select_model = view.selectionModel()
                select_flag = QtCore.QItemSelectionModel.ClearAndSelect
                #select_flag = QtCore.QItemSelectionModel.Select
                #select_flag = QtCore.QItemSelectionModel.NoUpdate
                with ut.Timer('[api_item_view] selecting name. qtindex=%r' % (qtindex,)):
                    select_model.select(qtindex, select_flag)
                with ut.Timer('[api_item_view] expanding'):
                    view.setExpanded(qtindex, True)
            else:
                # For Table Views
                view.selectRow(row)
            # Scroll to selection
            if scroll:
                with ut.Timer('scrolling'):
                    view.scrollTo(qtindex)
            return row
    return None

If the user has manually scrolled past the row in question then this function works. However, if the user has not seen the specific row this function just scrolls back to the top of the view.

Answer 1

It's probably too late for the answer here but maybe it would still benefit someone in future.

Below one can find a working example of a list model with canFetchMore and fetchMore methods + a view with a couple of custom methods:

1. Method trying to load more items from the model, if the model has something not loaded yet
2. Method capable of fetching the specific rows from the model if they haven't been loaded yet

The QMainWindow subclass in the example has a timer which is used to repeatedly call the first of the above mentioned methods, each time forcing the load of another batch of items from the model into the view. The loading of items in batches over small time intervals allows one to avoid blocking the UI thread completely and be able to edit the items loaded so far with little to no lag. The example contains a progress bar showing the part of items loaded so far.

The QMainWindow subclass also has a spin box which allows one to pick a particular row to show in the view. If the corresponding item has already been fetched from the model, the view simply scrolls to it. Otherwise it fetches this row's item from the model first, in a synchronous i.e. UI blocking fashion.

Here's the full code of the solution, tested with python 3.5.2 and PyQt5:

import sys
from PyQt5 import QtWidgets, QtCore

class DelayedFetchingListModel(QtCore.QAbstractListModel):
    def __init__(self, batch_size=100, max_num_nodes=1000):
        QtCore.QAbstractListModel.__init__(self)
        self.batch_size = batch_size
        self.nodes = []
        for i in range(0, self.batch_size):
            self.nodes.append('node ' + str(i))
        self.max_num_nodes = max(self.batch_size, max_num_nodes)

    def flags(self, index):
        if not index.isValid():
            return QtCore.Qt.ItemIsEnabled
        return QtCore.Qt.ItemIsEnabled | QtCore.Qt.ItemIsSelectable | QtCore.Qt.ItemIsEditable;

    def rowCount(self, index):
        if index.isValid():
            return 0
        return len(self.nodes)

    def data(self, index, role):
        if not index.isValid():
            return None
        if role != QtCore.Qt.DisplayRole:
            return None
        row = index.row()
        if row < 0 or row >= len(self.nodes):
            return None
        else:
            return self.nodes[row]

    def setData(self, index, value, role):
        if not index.isValid():
            return False
        if role != QtCore.Qt.EditRole:
            return False
        row = index.row()
        if row < 0 or row >= len(self.nodes):
            return False
        self.nodes[row] = value
        self.dataChanged.emit(index, index)
        return True

    def headerData(self, section, orientation, role):
        if section != QtCore.Qt.Horizontal:
            return None
        if section != 0:
            return None
        if role != QtCore.Qt.DisplayRole:
            return None
        return 'node'

    def canFetchMore(self, index):
        if index.isValid():
            return False
        return (len(self.nodes) < self.max_num_nodes)

    def fetchMore(self, index):
        if index.isValid():
            return
        current_len = len(self.nodes)
        target_len = min(current_len + self.batch_size, self.max_num_nodes)
        self.beginInsertRows(index, current_len, target_len - 1)
        for i in range(current_len, target_len):
            self.nodes.append('node ' + str(i))
        self.endInsertRows()

class ListView(QtWidgets.QListView):
    def __init__(self, parent=None):
        QtWidgets.QListView.__init__(self, parent)

    def jumpToRow(self, row):
        model = self.model()
        if model == None:
            return False
        num_rows = model.rowCount()
        while(row >= num_rows):
            res = fetchMoreRows(QtCore.QModelIndex())
            if res == False:
                return False
            num_rows = model.rowCount()
        index = model.index(row, 0, QtCore.QModelIndex())
        self.scrollTo(index, QtCore.QAbstractItemView.PositionAtCenter)
        return True

    def fetchMoreRows(self, index):
        model = self.model()
        if model == None:
            return False
        if not model.canFetchMore(index):
            return False
        model.fetchMore(index)
        return True

class MainForm(QtWidgets.QMainWindow):
    def __init__(self, parent=None):
        QtWidgets.QMainWindow.__init__(self, parent)
        # Setup the model
        self.max_num_nodes = 10000
        self.batch_size = 100
        self.model = DelayedFetchingListModel(batch_size=self.batch_size, max_num_nodes=self.max_num_nodes)
        # Setup the view
        self.view = ListView()
        self.view.setModel(self.model)
        # Update the currently selected row in the spinbox
        self.view.selectionModel().currentChanged.connect(self.onCurrentItemChanged)
        # Select the first row in the model
        index = self.model.index(0, 0, QtCore.QModelIndex())
        self.view.selectionModel().clearSelection()
        self.view.selectionModel().select(index, QtCore.QItemSelectionModel.Select)
        # Setup the spinbox
        self.spinBox = QtWidgets.QSpinBox()
        self.spinBox.setMinimum(0)
        self.spinBox.setMaximum(self.max_num_nodes-1)
        self.spinBox.setSingleStep(1)
        self.spinBox.valueChanged.connect(self.onSpinBoxNewValue)
        # Setup the progress bar showing the status of model data loading
        self.progressBar = QtWidgets.QProgressBar()
        self.progressBar.setRange(0, self.max_num_nodes)
        self.progressBar.setValue(0)
        self.progressBar.valueChanged.connect(self.onProgressBarValueChanged)
        # Add status bar but initially hidden, will only show it if there's something to say
        self.statusBar = QtWidgets.QStatusBar()
        self.statusBar.hide()
        # Collect all this stuff into a vertical layout
        self.layout = QtWidgets.QVBoxLayout()
        self.layout.addWidget(self.view)
        self.layout.addWidget(self.spinBox)
        self.layout.addWidget(self.progressBar)
        self.layout.addWidget(self.statusBar)
        self.window = QtWidgets.QWidget()
        self.window.setLayout(self.layout)
        self.setCentralWidget(self.window)
        # Setup timer to fetch more data from the model over small time intervals
        self.timer = QtCore.QBasicTimer()
        self.timerPeriod = 1000
        self.timer.start(self.timerPeriod, self)

    def onCurrentItemChanged(self, current, previous):
        if not current.isValid():
            return
        row = current.row()
        self.spinBox.setValue(row)

    def onSpinBoxNewValue(self, value):
        try:
            value_int = int(value)
        except ValueError:
            return
        num_rows = self.model.rowCount(QtCore.QModelIndex())
        if value_int >= num_rows:
            # There is no such row within the model yet, trying to fetch more
            while(True):
                res = self.view.fetchMoreRows(QtCore.QModelIndex())
                if res == False:
                    # We shouldn't really get here in this example since out
                    # spinbox's range is limited by exactly the number of items
                    # possible to fetch but generally it's a good idea to handle
                    # cases like this, when someone requests more rows than 
                    # the model has
                    self.statusBar.show()
                    self.statusBar.showMessage("Can't jump to row %d, the model has only %d rows" % (value_int, self.model.rowCount(QtCore.QModelIndex())))
                    return
                num_rows = self.model.rowCount(QtCore.QModelIndex())
                if value_int < num_rows:
                    break;
        if num_rows < self.max_num_nodes:
            # If there are still items to fetch more, check if we need to update the progress bar
            if self.progressBar.value() < value_int:
                self.progressBar.setValue(value_int)
        elif num_rows == self.max_num_nodes:
            # All items are loaded, nothing to fetch more -> no need for the progress bar
            self.progressBar.hide()
        # Update the selection accordingly with the new row and scroll to it
        index = self.model.index(value_int, 0, QtCore.QModelIndex())
        selectionModel = self.view.selectionModel()
        selectionModel.clearSelection()
        selectionModel.select(index, QtCore.QItemSelectionModel.Select)
        self.view.scrollTo(index, QtWidgets.QAbstractItemView.PositionAtCenter)
        # Ensure the status bar is hidden now
        self.statusBar.hide()

    def timerEvent(self, event):
        res = self.view.fetchMoreRows(QtCore.QModelIndex())
        if res == False:
            self.timer.stop()
        else:
            self.progressBar.setValue(self.model.rowCount(QtCore.QModelIndex()))
            if not self.timer.isActive():
                self.timer.start(self.timerPeriod, self)

    def onProgressBarValueChanged(self, value):
        if value >= self.max_num_nodes:
            self.progressBar.hide()

def main():
    app = QtWidgets.QApplication(sys.argv)
    form = MainForm()
    form.show()
    app.exec_()

if __name__ == '__main__':
    main()

One more thing I'd like to note is that this example expects the fetchMore method to do its work synchronously. But in more sophisticated approaches fetchMore doesn't actually have to act so. If your model loads its items from, say, a database then talking with the database synchronously in the UI thread would be a bad idea. Instead fetchMore implementation could start the asynchronous sequence of signal/slot communications with some object handling the communication with the database occurring in some background thread.