Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to find crc32 of big files?

The PHP's crc32 support string as input.And For a file , below code will work OFC.

crc32(file_get_contents("myfile.CSV"));

But if file goes huge (2 GB) it might raise out of memory Fatal error.

So any way around to find checksum of huge files ?

like image 462
Arshdeep Avatar asked Jun 05 '10 18:06

Arshdeep


People also ask

How do I find the crc32 of a file?

Right-click the file you wish to get the CRC-32 for. A context menu appears. Select the CRC SHA submenu entry. Select any of the available algorithms: CRC-32, CRC-64, SHA-1 or SHA-256 to calculate the respective checksum, or select "*" to calculate all of them and additionally BLAKE2sp.

How big is crc32?

The CRC32 function returns an 8-character string that is a text representation of the hexadecimal value of a 32-bit binary sequence.

What is a crc32 file?

DESCRIPTION. crc32 is a simple utility that calculates the CRC-32 checksum for each of the given files. Note that the CRC-32 checksum is merely used for error detection in transmission and storage. It is not intended to guard against the malicious modification of files (i.e., it is not a cryptographic hash).

How many bits is crc16?

The most commonly used polynomial lengths are 9 bits (CRC-8), 17 bits (CRC-16), 33 bits (CRC-32), and 65 bits (CRC-64). A CRC is called an n-bit CRC when its check value is n-bits. For a given n, multiple CRCs are possible, each with a different polynomial.


2 Answers

PHP doesn't support files larger than 2GB (32bit limitation)

And more efficient way to calculate crc32 from files:

$hash = hash_file('crc32b',"myfile.CSV" );
like image 126
dev-null-dweller Avatar answered Oct 27 '22 00:10

dev-null-dweller


This function in the User Contributed Notes to crc32() claims to calculate the value without loading the file in full. If it works correctly, it should eliminate any memory problems.

For a file larger than 2 GB, it is however likely to stop at the same 32-bit limitation you are encountering right now.

If possible, I would invoke an external tool that can calculate the checksum for files as large as the one at hand.

like image 33
Pekka Avatar answered Oct 27 '22 00:10

Pekka