I see this a lot and haven't figured out a graceful solution. If user input contains invalid byte sequences, I need to be able to have it not raise an exception. For example:
# @raw_response comes from user and contains invalid UTF-8 # for example: @raw_response = "\xBF" regex.match(@raw_response) ArgumentError: invalid byte sequence in UTF-8
Numerous similar questions have been asked and the result appears to be encoding or force encoding the string. Neither of these work for me however:
regex.match(@raw_response.force_encoding("UTF-8")) ArgumentError: invalid byte sequence in UTF-8
or
regex.match(@raw_response.encode("UTF-8", :invalid=>:replace, :replace=>"?")) ArgumentError: invalid byte sequence in UTF-8
Is this a bug with Ruby 2.0.0 or am I missing something?
What is strange is it appear to be encoding correctly, but match continues to raise an exception:
@raw_response.encode("UTF-8", :invalid=>:replace, :replace=>"?").encoding => #<Encoding:UTF-8>
String literals are simply strings that are created using Ruby's built-in string syntax. This includes those created using double quotes (“), single quotes ('), or the special forms %Q and %q. Traditionally, these forms have all created strings that can be modified.
In Ruby 2.0 the encode
method is a no-op when encoding a string to its current encoding:
Please note that conversion from an encoding
enc
to the same encodingenc
is a no-op, i.e. the receiver is returned without any changes, and no exceptions are raised, even if there are invalid bytes.
This changed in 2.1, which also added the scrub
method as an easier way to do this.
If you are unable to upgrade to 2.1, you’ll have to encode into a different encoding and back in order to remove invalid bytes, something like:
if ! s.valid_encoding? s = s.encode("UTF-16be", :invalid=>:replace, :replace=>"?").encode('UTF-8') end
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With