Converting an xml doc into a specific dot-expanded json structure

Question

I have the following XML document:

<Item ID="288917">
  <Main>
    <Platform>iTunes</Platform>
    <PlatformID>353736518</PlatformID>
  </Main>
  <Genres>
    <Genre FacebookID="6003161475030">Comedy</Genre>
    <Genre FacebookID="6003172932634">TV-Show</Genre>
  </Genres>
  <Products>
    <Product Country="CA">
      <URL>https://itunes.apple.com/ca/tv-season/id353187108?i=353736518</URL>
      <Offers>
        <Offer Type="HDBUY">
          <Price>3.49</Price>
          <Currency>CAD</Currency>
        </Offer>
        <Offer Type="SDBUY">
          <Price>2.49</Price>
          <Currency>CAD</Currency>
        </Offer>
      </Offers>
    </Product>
    <Product Country="FR">
      <URL>https://itunes.apple.com/fr/tv-season/id353187108?i=353736518</URL>
      <Rating>Tout public</Rating>
      <Offers>
        <Offer Type="HDBUY">
          <Price>2.49</Price>
          <Currency>EUR</Currency>
        </Offer>
        <Offer Type="SDBUY">
          <Price>1.99</Price>
          <Currency>EUR</Currency>
        </Offer>
      </Offers>
    </Product>
  </Products>
</Item>

Currently, to get it into json format I'm doing the following:

parser = etree.XMLParser(recover=True)
node = etree.fromstring(s, parser=parser)
data = xmltodict.parse(etree.tostring(node))

Of course the xmltodict is doing the heavy lifting. However, it gives me a format that is not ideal for what I'm trying to accomplish. Here is what I'd like the end data to look like:

{
    "Item[@ID]": 288917, # if no preceding element, use the root node tag
    "Main.Platform": "iTunes",
    "Main.PlatformID": "353736518",
    "Genres.Genre": ["Comedy", "TV-Show"] # list of elements if repeated
    "Genres.Genre[@FacebookID]": ["6003161475030", "6003161475030"],
    "Products.Product[@Country]": ["CA", "FR"],
    "Products.Product.URL": ["https://itunes.apple.com/ca/tv-season/id353187108?i=353736518", "https://itunes.apple.com/fr/tv-season/id353187108?i=353736518"],
    "Products.Product.Offers.Offer[@Type]": ["HDBUY", "SDBUY", "HDBUY", "SDBUY"],
    "Products.Product.Offers.Offer.Price": ["3.49", "2.49", "2.49", "1.99"],
    "Products.Product.Offers.Offer.Currency": "EUR"    
}

Answer 1

This is a bit verbose, but it wasn't too hard to format this as a flat dict. Here is an example:

node = etree.fromstring(file_data.encode('utf-8'), parser=parser)
data = OrderedDict()
nodes = [(node, ''),] # format is (node, prefix)

while nodes:

    for sub, prefix in nodes:

        # remove the prefix tag unless its for the first attribute
        tag_prefix = '.'.join(prefix.split('.')[1:]) if ('.' in prefix) else ''
        atr_prefix = sub.tag if (sub == node) else tag_prefix

        # tag
        if sub.text.strip():
            _prefix = tag_prefix + '.' + sub.tag
            _value = sub.text.strip()
            if data.get(_prefix): # convert it to a list if multiple values
                if not isinstance(data[_prefix], list): data[_prefix] = [data[_prefix],]
                data[_prefix].append(_value)
            else:
                data[_prefix] = _value

        # atr
        for k, v in sub.attrib.items():
            _prefix = atr_prefix + '[@%s]' % k
            _value = v
            if data.get(_prefix): # convert it to a list if multiple values
                if not isinstance(data[_prefix], list): data[_prefix] = [data[_prefix],]
                data[_prefix].append(_value)
            else:
                data[_prefix] = _value

        nodes.remove((sub, prefix))

        for s in sub.getchildren():
            _prefix = (prefix + '.' + sub.tag).strip('.')
            nodes.append((s, _prefix))

    if not nodes: break