How can I split a text file based on comment blocks in Python?

Question

I wasted most of my morning failing to solve this simple problem. Using python, I want to parse data files that look like this:

# This is an example comment line, it starts with a '#' character.
# There can be a variable number of comments between each data set.
# Comments "go with" the data set that comes after them.
# The first data set starts on the next line:
0.0 1.0
1.0 2.0
2.0 3.0
3.0 4.0

# Data sets are followed by variable amounts of white space.
# The second data set starts after this comment
5.0 6.0
6.0 7.0


# One more data set.
7.0 8.0
8.0 9.0

The python code I want would parse the above example into the three "blocks", storing them as elements of a list. The individual code-blocks could themselves be stored as lists of lines, with or without the comment lines, whatever. A handraulic way is to do this:

#! /usr/bin/env python

# Read in data, seperate into rows_alldata
f=open("example")
rows = f.read().split('\n')
f.close()

# Do you haz teh codez?
datasets=[]
datasets.append(rows[0:8])
datasets.append(rows[9:13])
datasets.append(rows[15:18])

I'm looking for a more general solution that supports variable numbers and lengths of data sets. I have tried several catastrophes built off non-pythonic looking loops. I think it best not to clutter up my question with them; this is work and not "homework".

Answer 1

Use groupby.

from itertools import groupby

def contains_data(ln):
    # just an example; there are smarter ways to do this
    return ln[0] not in "#\n"

with open("example") as f:
    datasets = [[ln.split() for ln in group]
                for has_data, group in groupby(f, contains_data)
                if has_data]

Answer 2

datasets = [[]]
with open('/tmp/spam.txt') as f:
  for line in f:
    if line.startswith('#'):
      if datasets[-1] != []:
        # we are in a new block
        datasets.append([])
    else:
      stripped_line = line.strip()
      if stripped_line:
        datasets[-1].append(stripped_line)

Answer 3

import pprint

with open("test.txt") as fh:
    codes = []
    codeblock = []

    for line in fh:
        stripped_line = line.strip()

        if not stripped_line:
            continue

        if stripped_line.startswith("#"):
            if codeblock:
                codes.append(codeblock)
                codeblock = []

        else:
            codeblock.append(stripped_line.split(" "))

    if codeblock:
        codes.append(codeblock)

pprint.pprint(codes)

Output:

[[['0.0', '1.0'], ['1.0', '2.0'], ['2.0', '3.0'], ['3.0', '4.0']],
 [['5.0', '6.0'], ['6.0', '7.0']],
 [['7.0', '8.0'], ['8.0', '9.0']]]