I am trying to create a patch using two large size folders (~7GB).
Here is how I'm doing it :
$ diff -Naurbw . ../other-folder > file.patch
But maybe due to file sizes, patch is not getting created and giving an error:
diff: memory exhausted
I tried making space more than 15 GB but still the issue persists. Could someone help me out with the flags that I should use?
bsdiff
is slow & requires large memory, xdelta
is create large delta for large files.
Try HDiffPatch for large files: https://github.com/sisong/HDiffPatch
Usage example:
hdiffz -s-256 [-c-lzma2] old_path new_path out_delta_file
hpatchz old_path delta_file out_new_path
Recently I came across this too when I needed to diff two large files (>5Gb each).
I tried to use 'diff' with different options, but even the --speed-large-files
had no effect. Other methods like splitting the files into smaller ones, using xdelta
or sorting the files as per this suggestion didn't help either. I even got my hands around a very powerful VM (> 72Gb RAM), but still got this memory exhausted
error.
I finally got to work by adding the following parameter to sysctl.conf
(sudo vim /etc/sysctl.conf
):
vm.overcommit_memory=1
vm.overcommit_memory
has three values (0,1,2) and sets the kernel virtual memory accounting mode. From the proc(5) man page:
0: heuristic overcommit (this is the default)
1: always overcommit, never check
2: always check, never overcommit
To make sure that the parameter is indeed applied you can run
sudo sysctl -p
Don't forget to change this parameter back when you finish!
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With