Apache Spark UI displays incorrect input size of file being ingested

Question

My Java spark program ingests a file of 3.7 GB. When I launch the spark program and go to the Spark UI on port localhost:4040 The input size shown for the load stage is 7.3 GB??? That's really confusing. Why is the input size in the Spark UI console showing almost double than the actual file size being ingested?

Answer 1

The input size:

• Is estimated.
• Is not the input size of the file you load, but the input size of the loaded object, which in general, require more memory to store than a serialized objects (pointers to actual objects, overhead of the data structures used to load the data).