Laravel: Unable to JSON encode payload. Error code: 5

Question

I'm working on Laravel (v5.7) app that converts uploaded CSV (with contacts) into array that is then passed as argument when job class is being dispatched.

Here is the example of CSV file (format that is supported):

123456,Richard,Smith
654321,John,Doe

Uploaded (CSV) file is handled like this:

$file_path = $request->file_name->store('contacts');
$file = storage_path('app/' . $file_path);

$contactsIterator = $this->getContacts($file);

$contacts = iterator_to_array($contactsIterator); // Array of contacts from uploaded CSV file

    protected function getContacts($file)
    {
        $f = fopen($file, 'r');

        while ($line = fgets($f))
        {
            $row = explode(",", $line);

            yield [
                'phone'     => !empty($row[0]) ? trim($row[0]) : '',
                'firstname' => !empty($row[1]) ? trim($row[1]) : '',
                'lastname'  => !empty($row[2]) ? trim($row[2]) : '',
            ];
        }
    }

Finally, $contacts array is passed to a job that is dispatched:

ImportContacts::dispatch($contacts);

This job class looks like this:

    public function __construct($contacts)
    {
        Log::info('ImportContacts@__construct START');
        $this->contacts = $contacts;
        Log::info('ImportContacts@__construct END');
    }

    public function handle()
    {
        Log::info('ImportContacts@handle');
    }

... and everything worked fine (no errors) until I've tried with this CSV:

123456,Richardÿ,Smith
654321,John,Doe

Please notice ÿ. So, when I try with this CSV - I get this error exception:

/code_smsto/vendor/laravel/framework/src/Illuminate/Queue/Queue.php | 91 | Unable to JSON encode payload. Error code: 5 

... and my log file looks like this:

  error local   2019-11-11 17:17:18     /code_smsto/vendor/laravel/framework/src/Illuminate/Queue/Queue.php | 91 | Unable to JSON encode payload. Error code: 5
  info  local   2019-11-11 17:17:18     ImportContacts@__construct END
  info  local   2019-11-11 17:17:18     ImportContacts@__construct START 

As you can see - handle method was never executed. If I remove ÿ - no errors and handle is executed.

I've tried to solve this, but without success:

1. Apply utf8_encode:

    protected function getContacts($file, $listId)
    {
        $f = fopen($file, 'r');

        while ($line = fgets($f))
        {
            $row = explode(",", $line);

            yield [
                'phone'     => !empty($row[0]) ? utf8_encode($row[0]) : '',
                'firstname' => !empty($row[1]) ? utf8_encode($row[1]) : '',
                'lastname'  => !empty($row[2]) ? utf8_encode($row[2]) : '',
            ];
        }
    }

... and it works (no errors, no matter if there's that ÿ), but then Greek and Cyrillic letters are turned into question marks. For example, this: Εθνικής will become ???????.

I also tried with mb_convert_encoding($row[1], 'utf-8') - and it doesn't turn Greek or Cyrillic letter into question marks, but this ÿ character will become ?.

2. Move "handling" (converting to array) of uploaded CSV file into @handle method of a Job class worked, but then I was not able to store the data from that array into DB (MongoDB). Please see the update below.

---

DEBUGGING:

This is what I get from dd($contacts);:

So, it has that "b" where ÿ is. And, after some "googling" I found that this "b" means "binary string", that is, a non unicode string, on which functions operate at the byte level (What does the b in front of string literals do?).

What I understand is this: When dispatching Job class, Laravel tries to "JSON encode" it (passed arguments/data) but it fails because there are binary data (non-unicode strings). Anyway, I was not able to find a solution (to be able to handle such CSV file with ÿ).

I am using:

• Laravel 5.7
• PHP 7.1.31-1+ubuntu16.04.1+deb.sury.org+1 (cli) (built: Aug 7 2019 10:22:48) ( NTS )
• Redis powered queues

---

UPDATE

When I move "handling" (converting to array) of uploaded CSV file into @handle method of a Job class - I don't get this error (Unable to JSON encode payload. Error code: 5), but when I try to store that problematic binary data with ÿ (b"Richardÿ") into MongoDB - it fails. The weird thing is that I don't get any error-exception message in log file, so I put all in try-catch like this:

        try {
            // Insert data into MongoDB
        } catch (Exception $e) {
            Log::info($e->getFile());
            Log::info($e->getLine());
            Log::info($e->getMessage());
        }

... and this is the result:

Anyway, I believe that it failed because of b"Richardÿ", and I guess that the solution is in encoding string, but as I've mentioned - I was not able to find a solution that works:

• utf8_encode works (no errors, no matter if there's that ÿ), but then Greek and Cyrillic letters are turned into question marks. For example, this: Εθνικής will become ???????
• mb_convert_encoding($row[1], 'utf-8') - it doesn't turn Greek or Cyrillic letter into question marks, but this ÿ character will become ?.
• iconv('windows-1252', 'UTF-8', $row[1]) - works (no errors, no matter if there's that ÿ), but when there are Greek or Cyrillic letters - it fails (I get this error exception: iconv(): Detected an illegal character in input string)

Answer 1

You have several ways to deal with it but I'd recommend the following two. In both cases, the idea is that you store a UTF-8 string.

A simpler approach, figure out what encoding it is out of the (your) predefined list and convert it to UTF8.

$encoding = mb_detect_encoding($content, 'UTF-8, ISO-8859-1, WINDOWS-1252, WINDOWS-1251', true);
if ($encoding != 'UTF-8') {
    $string = iconv($encoding, 'UTF-8//IGNORE', $row[1]);
}

The second approach is to use a third party library outlined in this answer