Finding first and last rows in Pandas Dataframes for individual files

Question

I have a Pandas Dataframe consisting of multiple .fits files, each one containing multiple columns with individual labels. I'd like to extract one column and create variables that contain the first and last rows of said column but I'm having a hard time accomplishing that for the individual .fits files and not just the entire Dataframe. Any help would be appreciated! :)

Here is how I read in my files:

path = '/Users/myname/folder/'
m = [os.path.join(dirpath, f)
    for dirpath, dirnames, files in os.walk(path)
    for f in fnmatch.filter(files, '*.fits')]

^^^ This recursively searches through my directory containing multiple .fits files in many subfolders.

dataframes = []
for ii in range(0,len(m)):
    data = pd.read_csv(m[ii], header = 'infer', delimiter = '\t')
    d = pd.DataFrame(data)
    top = d['desired_column'].head()
    bottom = d['desired_column'].tail()
    First_and_Last = pd.concat([top,bottom])

I tried using the .head and .tail commands for Pandas Dataframes but I am unsure how to properly use it for what I desire. For how I read in my fits files, the following code gives me the very first few rows and the very last few rows (5 to be exact with the default value for head and tail being 5) as seen here:

0       2.456849e+06
1       2.456849e+06
2       2.456849e+06
3       2.456849e+06
4       2.456849e+06
1118    2.456852e+06
1119    2.456852e+06
1120    2.456852e+06
1121    2.456852e+06
1122    2.456852e+06

What I want to do is try to get the first and last row for each .fits file for the specific column I want and not just for the Dataframe containing the .fits files. With the way I am reading in my .fits files, the Dataframe seems to sort of concatenate all the files together. Any tips on how I can accomplish this goal?

Answer 1

If you want only the first row:

top = d['desired_column'].head(1)

If you want only the last row:

bottom = d['desired_column'].tail(1)

I didn't find the problem of "Dataframe seems to sort of concatenate all the files together." Would you please clarify the question?
Btw, after data = pd.read_csv(m[ii], header = 'infer', delimiter = '\t'), data is already a DataFrame. Therefore, d = pd.DataFrame(data) is unnecessary.