Does groupby in pandas create a copy of the data or just a view?

Question

Does pandas.DataFrame.groupby create a copy of the data or just a view? In the (more probable) case of not creating a copy, what is the additional memory overhead and how does it scale with the original dataframe chracteristics (e.g. number of rows, columns, distinct groups)?

Answer 1

I did a little more research on this since someone asked me to help them with this question, and the pandas source code has been revised somewhat since the accepted answer was written.

According to what I can tell from the source code:

Groupby returns the groups on a Grouper object (i.e. Grouper.groups), which are “a specification for a groupby instruction”.

Ok, so what does that mean?

“Groupers are ultimately index mappings.”

I've always thought of this as meaning that groupby is creating a new object. It's not a full copy of the original dataframe, because you're performing selections and aggregations. So it's more like a transformation in that sense.

If your definition of a view is like this: "A view is nothing more than a SQL statement that is stored in the database with an associated name. A view is actually a composition of a table in the form of a predefined SQL query", then I'm wondering if what you're really asking is whether the groupby operation has to be re-applied each time you execute the same grouping on the same dataframe?

If that's what you're asking, I'd say the answer is no, it's not like a view, as long as you store the result of the grouping operation. The output object of a grouped dataframe or series is a (new) dataframe or series.