Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Pandas Get a List Of All Data Frames loaded into memory

Tags:

python

pandas

I am using pandas to read several csv files into memory for processing and at some point would like to list all the data frames I have loaded into memory. Is there a simple way to do that? (I am thinking something like %ls but only for the data frames that I have available in memory)

like image 623
Kartik Avatar asked Dec 13 '16 04:12

Kartik


People also ask

How do I display all Dataframes in Python?

Method 2: Using set_option() display. max_rows represents the maximum number of rows that pandas will display while displaying a data frame. The default value of max_rows is 10. If set to 'None' then it means all rows of the data frame.

Can you have a list of Dataframes?

Creating a list of Dataframes. To create a list of Dataframes we use the list() function in R and then pass each of the data frame you have created as arguments to the function.

Are pandas Dataframes stored in memory?

You can work with datasets that are much larger than memory, as long as each partition (a regular pandas DataFrame) fits in memory.


3 Answers

I personally think this approach is much better (if in ipython).

import pandas as pd
%whos DataFrame
like image 114
Alex Avatar answered Oct 07 '22 10:10

Alex


You could list all dataframes with the following:

import pandas as pd

# create dummy dataframes
df1 = pd.DataFrame({'Col1' : list(range(100))})
df2 = pd.DataFrame({'Col1' : list(range(100))})

# check whether all variables in scope are pandas dataframe. 
# Dir() will return a list of string representations of the variables. 
# Simply evaluate and test whether they are pandas dataframes
alldfs = [var for var in dir() if isinstance(eval(var), pd.core.frame.DataFrame)]

print(alldfs) # df1, df2
like image 27
datawrestler Avatar answered Oct 07 '22 08:10

datawrestler


building on previous answers ... this returns a list

import pandas as pd 
%who_ls DataFrame 

however, if you try to run a script it doesn't work

thus

import pandas as pd
sheets=[]    
for var in dir():
    if isinstance(locals()[var], pd.core.frame.DataFrame)  and var[0]!='_':
        sheets.append(var)

since some DataFrames will have a copy for internal use only and those start with '_'

like image 7
Jorge Martins Avatar answered Oct 07 '22 10:10

Jorge Martins