Reading GPS RINEX data with Pandas

Question

I am reading a [RINEX-3.02] (page 60) Observation Data file to do some timed based satellite ID filtering, and will eventually reconstruct it latter. This would give me more control over the selection of satellites I allow to contribute to a position solution over time with RTK post processing.

Specifically for this portion though, I'm just using:

• [python-3.3]
• [pandas]
• [numpy]

Here is an sample with the first three time stamped observations.
Note: It is not necessary for me to parse data from the header.

     3.02           OBSERVATION DATA    M: Mixed            RINEX VERSION / TYPE
CONVBIN 2.4.2                           20130731 223656 UTC PGM / RUN BY / DATE 
log: /home/ruffin/Documents/Data/in/FlagStaff_center/FlagStaCOMMENT             
format: u-blox                                              COMMENT             
                                                            MARKER NAME         
                                                            MARKER NUMBER       
                                                            MARKER TYPE         
                                                            OBSERVER / AGENCY   
                                                            REC # / TYPE / VERS 
                                                            ANT # / TYPE        
   808673.9171 -4086658.5368  4115497.9775                  APPROX POSITION XYZ 
        0.0000        0.0000        0.0000                  ANTENNA: DELTA H/E/N
G    4 C1C L1C D1C S1C                                      SYS / # / OBS TYPES 
R    4 C1C L1C D1C S1C                                      SYS / # / OBS TYPES 
S    4 C1C L1C D1C S1C                                      SYS / # / OBS TYPES 
  2013     7    28     0    27   28.8000000     GPS         TIME OF FIRST OBS   
  2013     7    28     0    43   43.4010000     GPS         TIME OF LAST OBS    
G                                                           SYS / PHASE SHIFT   
R                                                           SYS / PHASE SHIFT   
S                                                           SYS / PHASE SHIFT   
  0                                                         GLONASS SLOT / FRQ #
 C1C    0.000 C1P    0.000 C2C    0.000 C2P    0.000        GLONASS COD/PHS/BIS 
                                                            END OF HEADER       
> 2013  7 28  0 27 28.8000000  0 10                     
G10  20230413.601       76808.847       -1340.996          44.000  
G 4  20838211.591      171263.904       -2966.336          41.000  
G12  21468211.719      105537.443       -1832.417          43.000  
S38  38213212.070       69599.2942      -1212.899          45.000  
G 5  22123924.655     -106102.481        1822.942          46.000  
G25  23134484.916      -38928.221         656.698          40.000  
G17  23229864.981      232399.788       -4048.368          41.000  
G13  23968536.158        6424.1143       -123.907          28.000  
G23  24779333.279      103307.5703      -1805.165          29.000  
S35  39723655.125       69125.5242      -1209.970          44.000  
> 2013  7 28  0 27 29.0000000  0 10                     
G10  20230464.937       77077.031       -1341.254          44.000  
G 2  20684692.905       35114.399        -598.536          44.000  
G12  21468280.880      105903.885       -1832.592          43.000  
S38  38213258.255       69841.8772      -1212.593          45.000  
G 5  22123855.354     -106467.087        1823.084          46.000  
G25  23134460.075      -39059.618         657.331          40.000  
G17  23230018.654      233209.408       -4048.572          41.000  
G13  23968535.044        6449.0633       -123.060          28.000  
G23  24779402.809      103668.5933      -1804.973          29.000  
S35  39723700.845       69367.3942      -1208.954          44.000  
> 2013  7 28  0 27 29.2000000  0 9                     
G10  20230515.955       77345.295       -1341.436          44.000  
G12  21468350.548      106270.372       -1832.637          43.000  
S38  38213304.199       70084.4922      -1212.840          45.000  
G 5  22123786.091     -106831.642        1822.784          46.000  
G25  23134435.278      -39190.987         657.344          40.000  
G17  23230172.406      234019.092       -4048.079          41.000  
G13  23968534.775        6473.9923       -125.373          28.000  
G23  24779471.004      104029.6643      -1805.983          29.000  
S35  39723747.025       69609.2902      -1209.259          44.000  

If I do have to make a custom parser,
The other tricky thing is satellite IDs come and go over time,
(as shown with satellites "G 2" and "G 4")
(plus they have spaces in the IDs too)
So as I read them into a DataFrame,
I need to make new column labels (or row labels for MultiIndex?) as I find them.

I was initially thinking this could be considered a MultiIndex problem,
but I'm not so sure pandas read_csv could do everything
Jump to Reading DataFrame objects with MultiIndex

Any suggestions?

Relevant sources if interested:

• [python-3.3]: http://www.python.org/download/releases/3.3.0/
• [numpy]: http://www.numpy.org/
• [pandas]: http://pandas.pydata.org/
• [RINEX-3.02]: http://igscb.jpl.nasa.gov/igscb/data/format/rinex302.pdf
• [ephem]: https://pypi.python.org/pypi/ephem/
• [RTKLIB]: http://www.rtklib.com/
• [NOAA CORS]: http://geodesy.noaa.gov/CORS/

Answer 1

Here is what I ended up doing

df = readObs(indir, filename)
df.set_index(['%_GPST', 'satID'])

Note that I just set the new MultiIndex at the end after building it.

def readObs(dir, file):
    df = pd.DataFrame()
    #Grab header
    header = ''
    with open(dir + file) as handler:
        for i, line in enumerate(handler):
            header += line
            if 'END OF HEADER' in line:
                break
    #Grab Data
    with open(dir + file) as handler:
        for i, line in enumerate(handler):
            #Check for a Timestamp lable
            if '> ' in line:
                #Grab Timestamp
                links = line.split()
                index = datetime.strptime(' '.join(links[1:7]), '%Y %m %d %H %M %S.%f0')
                #Identify number of satellites
                satNum = int(links[8])
                #For every sat
                for j in range(satNum):
                    #just save the data as a string for now
                    satData = handler.readline()
                    #Fix the names
                    satdId = satData.replace("G ", "G0").split()[0]
                    #Make a dummy dataframe
                    dff = pd.DataFrame([[index,satdId,satData]], columns=['%_GPST','satID','satData'])
                    #Tack it on the end
                    df = df.append(dff)
    return df, header

Using a dummy data-frame just doesn't seem the most elegant though.

Answer 2

I suggest to write a custom parser, read the file line by line.

The space inebtween "G 5" is a further hint to write a custom parser.
In that case you cannot split the arguments simply by space,
you have to read all 3 chars at once, and remove the first char, and convert the remaining two (" 5") to a sattelite number.