How to get transferred size of a complete page load?

Question

With Selenium or JavaScript how could you get the (over the network) transferred size (bytes) of the loaded page including all the content, images, css, js, etc?

The preferred size is that of what goes over the network, that is compressed, only for the requests that are made, etc.

This is what you usually can see in dev tools, to the right in the network status bar:

 

If that's not possible, could one just get a total size of all the loaded resources (without compression, etc)? That would be an acceptable alternative.

The browser is Firefox, but if it could be done with some other Selenium compatible browser that would be acceptable also.

 

I guess this could be done using a proxy, but is there any JS or Selenium way to get such information?

If proxy is the only way, which one would one use (or implement) to keep things simple for such a task? Just implementing something in Java before setting up the driver?

(The solution should work at least on Linux, but preferably on Windows also. I'm using Selenium WebDriver via Java.)

Answer 1

For future reference, it is possible to request this information from the browser by javascript. However, at the time of writing no browser supports this feature for this specific data yet. More information can be found here.

In the mean time, for Chrome you can parse this information from the performance log.

    //Enable performance logging
    LoggingPreferences logPrefs = new LoggingPreferences();
    logPrefs.enable(LogType.PERFORMANCE, Level.ALL);
    capa.setCapability(CapabilityType.LOGGING_PREFS, logPrefs);

    //Start driver
    WebDriver driver = new ChromeDriver(capa);

You can then get this data like this

for (LogEntry entry : driver.manage().logs().get(LogType.PERFORMANCE)) {
        if(entry.getMessage().contains("Network.dataReceived")) {
            Matcher dataLengthMatcher = Pattern.compile("encodedDataLength\":(.*?),").matcher(entry.getMessage());
            dataLengthMatcher.find();
            //Do whatever you want with the data here.
        }

If, like in your case, you want to know the specifics of a single page load, you could use a pre- and postload timestamp and only get entries within that timeframe.

Answer 2

The performance API mentioned in Hakello's answer is now well supported (on everything except IE & Safari), and is simple to use:

return performance
  .getEntriesByType("resource")
  .map((x) => x.transferSize)
  .reduce((a, b) => (a + b), 0);

You can run that script using executeScript to get the number of bytes downloaded since the last navigation event. No setup or configuration is required.

Answer 3

Yes you can do it using BrowserMobProxy. This is a java jar which use selenium Proxy to track network traffic from client side. like page load time duration, Query string to different services etc. you can get it bmp.lightbody.net . This api will create .har files which will contain all these information in json format which you can read using an online tool http://www.softwareishard.com/har/viewer/