How to check if PDF is password protected using static tools

Question

I've seen multiple questions regarding this topic. In the answer, user have to load the pdf with some pdf lib or built in support and then based on library function, user can determine if the pdf is encrypted or not. I'm interested in knowing whether it is possible to detect PDF as encrypted using some static analysis tool i.e YARA where we are only reading the content of file as binary/string data instead of complete structure of the file?

Answer 1

PDF Encryption is indicated in the File Trailer dictionary with an Encrypt entry, per section 3.4.4 of the Adobe PDF (v.1.7) Reference manual (Table 3.13).

So to detect PDF Encryption, go to the end of the file and search upwards for the first line containing just the word 'trailer', then search downwards again for the string '/Encrypt'. If it's there, the file is encrypted, otherwise not.

Now, detecting whether a PDF is Password-protected, meaning so that you can't open it without supplying a password, is going to be harder. You basically need to read the Object reference after the /Encrypt key (e.g. '14 0 R'), jump to the beginning of the file and search for that object (e.g. '14 0 obj <<') and look for the /Filter , /R , and /U keys in that dictionary.

if the /Filter value is /Standard, then Per the preamble to Algorithm 3.6 'Authenticating the user password' (page 127), algorithm 3.6 can be used to determined whether the user password is the empty string and therefore whether to suppress prompting for a password.

So basically, if the /R value is 2, you will look for the /U value to be a specific string, and if the /R value is 3 or 4, you can look for the /U value to be another specific string, and if it is neither of those values then a user password is required to open the document and the document is password protected.

You could calculate those /U values by following the algorithms in the PDF Reference, or you can dig them out of existing encrypted PDFs that don't require a password to be opened.