QWebEngineView - loading of > 2mb content

Question

So, using PyQt5's QWebEngineView and the .setHTML and .setContent methods have a 2 MB size limitation. When googling for solutions around this, I found two methods:

Use SimpleHTTPServer to serve the file. This however gets nuked by a firewall employed in the company.

Use File Urls and point to local files. This however is a rather bad solution, as the HTML contains confidential data and I can't leave it on the harddrive, under any circumstance.

The best solution I currently see is to use file urls, and get rid of the file on program exit/when loadCompleted reports it is done, whichever comes first.

This is however not a great solution and I wanted to ask if there is a solution I'm overlooking that would be better?

Answer 1

Why don't you load/link most of the content through a custom url scheme handler?

webEngineView->page()->profile()->installUrlSchemeHandler("app", new UrlSchemeHandler(e));

class UrlSchemeHandler : public QWebEngineUrlSchemeHandler
{   Q_OBJECT
public:
    void requestStarted(QWebEngineUrlRequestJob *request) {
        QUrl url = request->requestUrl();
        QString filePath = url.path().mid(1);
        // get the data for this url
        QByteArray data = ..
        // 
        if (!data.isEmpty()) 
        {
            QMimeDatabase db;
            QString contentType = db.mimeTypeForFileNameAndData(filePath,data).name();
            QBuffer *buffer = new QBuffer();
            buffer->open(QIODevice::WriteOnly);
            buffer->write(data);
            buffer->close();
            connect(request, SIGNAL(destroyed()), buffer, SLOT(deleteLater()));
            request->reply(contentType.toUtf8(), buffer);
        } else {
            request->fail(QWebEngineUrlRequestJob::UrlNotFound);
        }
    }
};

you can then load a website by webEngineView->load(new QUrl("app://start.html"));

All relative pathes from inside will also be forwarded to your UrlSchemeHandler..

And rember to add the respective includes

#include <QWebEngineUrlRequestJob>
#include <QWebEngineUrlSchemeHandler>
#include <QBuffer>

Answer 2

One way you can go around this is to use requests and QWebEnginePage's method runJavaScript:

web_engine = QWebEngineView()
web_page = web_engine.page()
web_page.setHtml('')

url = 'https://youtube.com'
page_content = requests.get(url).text

# document.write writes a string of text to a document stream
# https://developer.mozilla.org/en-US/docs/Web/API/Document/write
# And backtick symbol(``) is for multiline strings
web_page.runJavaScript('document.write(`{}`);'.format(page_content))