os.walk() python: xml representation of a directory structure, recursion

Question

So I am trying to use os.walk() to generate an XML representation of a directory structure. I seem to be getting a ton of duplicates. It properly places directories within each other and files in the right place for the first portion of the xml file; however, after it does it correctly it then continues traversing incorrectly. I am not quite sure why....

Here is my code:

def dirToXML(self,directory):
        curdir = os.getcwd()
        os.chdir(directory)
        xmlOutput=""

        tree = os.walk(directory)
        for root, dirs, files in tree:
            pathName = string.split(directory, os.sep)
            xmlOutput+="<dir><name><![CDATA["+pathName.pop()+"]]></name>"
            if len(files)>0:
                xmlOutput+=self.fileToXML(files)
            for subdir in dirs:
                xmlOutput+=self.dirToXML(os.path.join(root,subdir))
            xmlOutput+="</dir>"

        os.chdir(curdir)
        return xmlOutput  

The fileToXML, simply parses out the list so no need to worry about that.

The Directory Structure is simply:

images/
images/testing.xml
images/structure.xml
images/Hellos
images/Goodbyes
images/Goodbyes/foo
images/Goodbyes/bar
images/Goodbyes/square

and the resulting xml file became:

<structure>
<dir>
<name>images</name>
  <files>
    <file>
      <name>structure.xml</name>
    </file>
    <file>
      <name>testing.xml</name>
    </file>
  </files>
  <dir>
    <name>Hellos</name>
  </dir>
  <dir>
    <name>Goodbyes</name>
    <dir>
      <name>foo</name>
    </dir>
    <dir>
      <name>bar</name>
    </dir>
    <dir>
      <name>square</name>
    </dir>
  </dir>
  <dir>
    <name>foo</name>
  </dir>
  <dir>
    <name>bar</name>
  </dir>
  <dir>
      <name>square</name>
    </dir>
  </dir>
  <dir>
    <name>Hellos</name>
  </dir>
  <dir>
    <name>Goodbyes</name>
    <dir>
      <name>foo</name>
    </dir>
    <dir>
      <name>bar</name>
    </dir>
    <dir>
      <name>square</name>
    </dir>
  </dir>
  <dir>
    <name>foo</name>
  </dir>
  <dir>
    <name>bar</name>
  </dir>
  <dir>
    <name>square</name>
  </dir>
</structure>

Any help would be much appreciated!

Answer 1

I'd recommend against using os.walk(), since you have to do so much to massage its output. Instead, just use a recursive function that uses os.listdir(), os.path.join(), os.path.isdir(), etc.

import os
from xml.sax.saxutils import escape as xml_escape

def DirAsXML(path):
    result = '<dir>\n<name>%s</name>\n' % xml_escape(os.path.basename(path))
    dirs = []
    files = []
    for item in os.listdir(path):
        itempath = os.path.join(path, item)
        if os.path.isdir(itempath):
            dirs.append(item)
        elif os.path.isfile(itempath):
            files.append(item)
    if files:
        result += '  <files>\n' \
            + '\n'.join('    <file>\n      <name>%s</name>\n    </file>'
            % xml_escape(f) for f in files) + '\n  </files>\n'
    if dirs:
        for d in dirs:
            x = DirAsXML(os.path.join(path, d))
            result += '\n'.join('  ' + line for line in x.split('\n'))
    result += '</dir>'
    return result

if __name__ == '__main__':
    print '<structure>\n' + DirAsXML(os.getcwd()) + '\n</structure>'

Personally, I'd recommend a much less verbose XML schema, putting names in attributes and getting rid of the <files> group:

import os
from xml.sax.saxutils import quoteattr as xml_quoteattr

def DirAsLessXML(path):
    result = '<dir name=%s>\n' % xml_quoteattr(os.path.basename(path))
    for item in os.listdir(path):
        itempath = os.path.join(path, item)
        if os.path.isdir(itempath):
            result += '\n'.join('  ' + line for line in 
                DirAsLessXML(os.path.join(path, item)).split('\n'))
        elif os.path.isfile(itempath):
            result += '  <file name=%s />\n' % xml_quoteattr(item)
    result += '</dir>'
    return result

if __name__ == '__main__':
    print '<structure>\n' + DirAsLessXML(os.getcwd()) + '\n</structure>'

This gives an output like:

<structure>
<dir name="local">
  <dir name=".hg">
    <file name="00changelog.i" />
    <file name="branch" />
    <file name="branch.cache" />
    <file name="dirstate" />
    <file name="hgrc" />
    <file name="requires" />
    <dir name="store">
      <file name="00changelog.i" />

etc.

If os.walk() worked more like expat's callbacks, you'd have an easier time of it.

Answer 2

Remove the two lines:

        for subdir in dirs:
            xmlOutput+=self.dirToXML(os.path.join(root,subdir))

you are recursing into the subdirectories; but that's redundant, because os.walk recurses itself.