When or how to use fetchMore() on QSqlTableModel with a SQLite database for rowCount() to work?

Question

My class DataTable is derived from QAbstractTableModel. It uses a QSqlTableModel object internally to fetch data from a db table. It represents a record for every row in the db (it does more but the record count is always the number of rows in the db table).

With MySql, my DataTable::rowCount() implementation just calls rowCount() on the QSqlTableModel, which works nicely.

Now with SQLite, Qt's SQLite driver returns a row count of 256 if there are more than 256 rows in the db table, so my DataTable class also returns 256 - which is wrong. The documentation tells me to call while (sql_model->canFetchMore()) sql_model->fetchMore();. Calling fetchMore() right after the internal QSqlTableModel is created actually causes the following rowCount() call to return the correct value. But as soon as something is changed in the database (my class would call insertRow() or setData() on the QSqlTableModel), the next QSqlTableModel::rowCount() call will again return 256.

The database is only modified by my class, which in turn uses that particular QSqlTableModel object (or a view, which uses my DataTable as model, could update something). So there's no other process that could insert rows into the database.

So when should my DataTable class call fetchMore() for rowCount() to always return the actual row count?
I'm thinking my class should connect some of the signals emitted by QSqlTableModel to a slot that would call fetchMore(), although I'm not sure if that's the proper/reliable way to do it?

---

Update:

Here's some code to demonstrate the basic issue.

QSqlTableModel *model = new QSqlTableModel(0, database); //QSqlDatabase
model->setTable("tablename");
qDebug() << "0 row count" << model->rowCount(); //0 row count 0 
model->setEditStrategy(QSqlTableModel::OnManualSubmit);
model->select();
qDebug() << "1 row count" << model->rowCount(); //1 row count 256 
while (model->canFetchMore()) model->fetchMore();
qDebug() << "2 row count" << model->rowCount(); //2 row count 1520 
//... other methods ...
model->setData(model->index(0, 0), "TEST");
model->submitAll();
qDebug() << "3 row count" << model->rowCount(); //3 row count 256 
while (model->canFetchMore()) model->fetchMore();
qDebug() << "4 row count" << model->rowCount(); //4 row count 1520 

After loading the sql model, rowCount() returns 256 (1), so fetchMore() has to be called. rowCount() then returns the actual row count.
Later, data is changed, after which rowCount() again returns 256 (3).

So it seems like fetchMore() has to be called after every write operation on the sql model. But rather than putting this while/canFetchMore()/fetchMore() loop at the end of every single method that modifies the model, I'm wondering if it would suffice to connect the beforeInsert(QSqlRecord&), beforeUpdate(int, QSqlRecord&) and beforeDelete(int) signals to a slot that would then call fetchAll()? Would this be reliable and appropriate?

Correction: Not before* signals (too early), but probably layoutChanged(), dataChanged(), rowsInserted() and rowsRemoved().

---

Update 2:

Note regarding SQL: I know that I could send a separate SELECT COUNT SQL query to the database in theory, but this doesn't answer the question. As long as I can avoid SQL, I won't write SQL. In my mind, sending such an SQL query defies the purpose of an object-oriented QAbstractTableModel class. Plus rowCount() is const (should not send queries) and should be fast. Anyway, this won't fix rowCount().

I ended up connecting a slot that calls fetchMore() to the relevant signals (see above) AND asserting that everything has been fetched in rowCount():
assert(!sql_model->canFetchMore())

This is because rowCount() not being able to report the correct row count counts as failure state to me, hence the assertion. In other words, I'd rather want my application to crash than use an incorrect row count.

Just connecting it to the dataChanged() signal (as suggested in the first answer: I would probably try to use dataChanged signal.) isn't sufficient. I have connected it to dataChanged(const QModelIndex&, const QModelIndex&), rowsInserted(const QModelIndex&, int, int), rowsRemoved(const QModelIndex&, int, int) and layoutChanged().

Seems to work, the assertion has not failed yet.

If someone could specifically confirm this (or explain why it won't always work), I'd appreciate an answer.

Answer 1

The dilemma you were facing was similar to what I encountered recently. I wrote a QT gui program which did the following--

i. Connect to and Query an Oracle Database ii. Show the query result in a QTableView iii. export the result of QTableView to a .csv file iv. import the .csv file to a loacl sqlite3 database v. Connect to and query the local sqlite3 database vi. perform query and show result in another QTableview

During Step ii., (i.e. export the result to .csv) what I observed was that although 543 records were getting generated on the QTableView, only 256 odd were getting exported to the .csv file.

I was using,

int rows=model->rowCount();
int columns=model->columnCount();

    for (int i = 0; i < rows; i++)
    {
        for (int j = 0; j < columns; j++)
        {
            textData += model->data(model->index(i,j)).toString();
            textData += ", ";      // for .csv file format
        }
        textData += "\n";             // (optional: for new line segmentation)
    }

    QFile csvfile("/home/aj/ora_exported.csv");
    if(csvfile.open(QIODevice::WriteOnly|QIODevice::Truncate))
    {
        QTextStream out(&csvfile);
        out<<textData;
    }
    csvfile.close();

as it turned out, the model read the model->rowcount before all the results were fetched.

So as suggested by SO community, I used---

while (model->canFetchMore())
       model->fetchMore();

    int rows=model->rowCount();
    int columns=model->columnCount();

    for (int i = 0; i < rows; i++)
    {
        for (int j = 0; j < columns; j++)
        {
            textData += model->data(model->index(i,j)).toString();
            textData += ", ";      // for .csv file format
        }
        textData += "\n";             // (optional: for new line segmentation)
    }

and all the records got populated (all 543) in the .csv file.

You can refer to my question at here.

Answer 2

From my experience, Qt's SQLite driver inserts rows into the model in steps of 256 rows. At the end of data fetching QSqlTableModel::rowCount() will return correct value, there is no need to call fetchMore(). It just depends where and when you are calling rowCount().

For example, if user is doing some calculations on rowsInserted() signal, his slot method will be called multiple times, and every time index of last row will be incremented for up to value of 256.

So, you need to use rowCount() in the end of data fetching (in some overridden QSqlTableModel methods, fetching is often not finished), or to not rely on intermediate row count value and to repeat calculations every time it changes.

This is a general answer, because you didn't post any code where we can see what are you exactly trying to do, but you can see where I am pointing to.

EDIT:

After QSqlTableModel::submitAll() call, model is repopulated and that is the reason why fetchMore() is needed to repeat again. Where to call fetchMore() depends on the use case. It could be called after the submitAll() or in some slot, there is no a general answer here. I would probably try to use dataChanged signal. However, the goal of fetching data always should be primary to display it in the view, and view does it on its own most of the cases.

In one of my applications, I was relying on a table view to fetch data for me. After setting model to table view, I either call QTableView::scrollToBottom() which fetches data for me (user should see latest data on the bottom anyway in my case, row count has a correct value after the scroll) or do calculations in rowsInserted() slot when user scrolls table, and data is again fetched automatically in steps of 256 rows.

And it is very important to know, if application needs to show somewhere number of rows in the table, or e.g. summary value of one column data, it is often much more efficient to use additional QSqlQuery to fetch additional information (like: select count(*) from xyz) then to read data from large table model. Table models are primary intended for providing data to table view.

Answer 3

The current answer does not fully answer the question (mentions dataChanged() signal but not the other signals), so I'm writing my own answer.

It's been a while and I believe I have covered all cases: I ended up connecting a slot that calls fetchMore() to the relevant signals AND asserting that everything has been fetched in my DataTable::rowCount() method: assert(!sql_model->canFetchMore())

(Of course rowCount() is a const method, so I couldn't fetch if nothing has been fetched yet, but that wouldn't be the job a getter anyway; the assertion is fine because canFetchMore() is also const.)

Signals: dataChanged(const QModelIndex&, const QModelIndex&), rowsInserted(const QModelIndex&, int, int), rowsRemoved(const QModelIndex&, int, int) and layoutChanged()

I'm using the assertion to make sure my model receives the correct row count or else the application would crash (which happens if not all of the mentioned signals are connected, for example layoutChanged()). This is important in my case, as an incorrect row count might lead to data loss in my case.

So far, the assertion hasn't failed, so I'm assuming this solves it.