Splitting very long (4GB) string with new lines

Question

I have a file that is supposed to be JSON objects, one per line. Unfortunately, a miscommunication happened with the creation of the file, and the JSON objects only have a space between them, not a new-line.

I need to fix this by replacing every instance of } { with }\n{.

Should be easy for sed or Perl, right?

sed -e "s/}\s{/}\n{/g" file.in > file.out

perl -pe "s/}\s{/}\n{/g" file.in > file.out

But file.in is actually 4.4 GB which seems to be causing a problem for both of these solutions.

The sed command finishes with a halfway-correct file, but file.out is only 335 MB and is only about the first 1/10th of the input file, cutting off in the middle of a line. It's almost like sed just quit in the middle of the stream. Maybe it's trying to load the entire 4.4 GB file into memory but running out of stack space at around 300MB and silently kills itself.

The Perl command errors with the following message:

[1] 2904 segmentation fault perl -pe "s/}\s{/}\n{/g" file.in > file.out

What else should I try?

Answer 1

Unlike the earlier solutions, this one handles {"x":"} {"}.

use strict;
use warnings;
use feature qw( say );

use JSON::XS qw( );

use constant READ_SIZE => 64*1024*1024;

my $j_in = JSON::XS->new->utf8;
my $j_out = JSON::XS->new;

binmode STDIN;
binmode STDOUT, ':encoding(UTF-8)';

while (1) {
   my $rv = sysread(\*STDIN, my $block, READ_SIZE);
   die($!) if !defined($rv);
   last if !$rv;

   $j_in->incr_parse($block);

   while (my $o = $j_in->incr_parse()) {
      say $j_out->encode($o);
   }
}

die("Bad data") if $j_in->incr_text !~ /^\s*\z/;

Answer 2

The default input record separator in Perl is \n, but you can change it to any character you want. For this problem, you could use { (octal 173).

perl -0173 -pe 's/}\s{/}\n{/g' file.in > file.out