Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Convert zero-padded bytes to UTF-8 string

I'm unpacking several structs that contain 's' type fields from C. The fields contain zero-padded UTF-8 strings handled by strncpy in the C code (note this function's vestigial behaviour). If I decode the bytes I get a unicode string with lots of NUL characters on the end.

>>> b'hiya\0\0\0'.decode('utf8')
'hiya\x00\x00\x00'

I was under the impression that trailing zero bytes were part of UTF-8 and would be dropped automatically.

What's the proper way to drop the zero bytes?

like image 435
Matt Joiner Avatar asked Feb 22 '11 04:02

Matt Joiner


3 Answers

Use str.rstrip() to remove the trailing NULs:

>>> 'hiya\0\0\0'.rstrip('\0')
'hiya'
like image 180
Adam Rosenfield Avatar answered Oct 16 '22 20:10

Adam Rosenfield


Either rstrip or replace will only work if the string is padded out to the end of the buffer with nulls. In practice the buffer may not have been initialised to null to begin with so you might get something like b'hiya\0x\0'.

If you know categorically 100% that the C code starts with a null initialised buffer and never never re-uses it, then you might find rstrip to be simpler, otherwise I'd go for the slightly messier but much safer:

>>> b'hiya\0x\0'.split(b'\0',1)[0]
b'hiya'

which treats the first null as a terminator.

like image 31
Duncan Avatar answered Oct 16 '22 20:10

Duncan


Unlike the split/partition-solution this does not copy several strings and might be faster for long bytearrays.

data = b'hiya\0\0\0'
i = data.find(b'\x00')
if i == -1:
  return data
return data[:i]
like image 3
phobie Avatar answered Oct 16 '22 18:10

phobie