Scapy: Adding new protocol with complex field groupings

Question

I'm trying to specify a new packet format using scapy. In the packet there is a list of items, and items consist of "grouped fields". By "grouped fields" I mean a sub-sequence of fields of different types. The only way of making "grouped fields" that I know of in scapy is by using Packet class and using FieldLenField/PacketListField to reference the length of the sequence and the type of list members. Is that the way to go? Something that looks like this:

from scapy.packet import Packet
from scapy.fields import *

class RepeatingGroupedSequence(Packet):
    name = "Simple group of two fields"

    fields_desc = [IntField('field1', 1), 
                   IntField('field2', 2)]

class TopLayer(Packet):
    name = "Storage for Repeating Sequence"

    fields_desc = [FieldLenField("length", None, count_of='rep_seq'),
                   PacketListField('rep_seq', None, RepeatingGroupedSequence, 
                                   count_from = lambda pkt: pkt.length),
                  ]

#Now here is the problem that I have with assembling PacketListField: 

#craft TopLayer packet
p = TopLayer()

#add two "repeated sequences"
p.rep_seq = [ RepeatingGroupedSequence(), RepeatingGroupedSequence() ]

#both sequences can observed
p.show()

#but the underlying structure of the repeated sequence is #Raw# at this stage
p.show2()

#length is 2
print p.rep_seq, 'length:', len(p.rep_seq)

#but the cloned packet has only one "repeated sequence", the rest is raw
clone = TopLayer(str(p))
clone.show()

#length is 1
print clone.rep_seq, 'length:', len(clone.rep_seq)

The problem with this approach is that the structure of the grouping is not preserved when the packet is reassembled. On assembly, the second instance of the RepeatedSequence is treated as a raw body, even though the count field is 2. How do you add RepeatingSequences like this so that structure is preserved on reassembly? Is there a way to group Fields without resorting to Packet as a storage type for lists?

Answer 1

Class RepeatingGroupedSequence needs to overwrite extract_padding method:

def extract_padding(self, s):
    return '', s

By default each sub packet treats everything as belonging to its own layer, ie:

def extract_padding(self, s):
    return s, None

And this is not what is used for grouping purposes. Can someone elaborate on the difference between padding and layer separation?