How to get csreq of macOS application on command line?

Question

I´m maintaining a github project which aims at automating clean macOS installations, macOS customization (complete system preferences and more settings) and and updates to apps and the system.

https://github.com/tiiiecherle/osx_install_config

One of the script sets the preferences under system preferences - securtiy - privacy.

https://github.com/tiiiecherle/osx_install_config/blob/master/11_system_and_app_preferences/11a_system_preferences_privacy_sqlite_mojave.sh

Every macOS app has a csreq which seems to be kind of a fingerprint / checksum for each app. Before macOS Mojave it was not needed to set the value explicitly and it was working to replace the csreq with a "?". For example allowing accessibility to the Terminal:

DATABASE_SYSTEM="/Library/Application Support/com.apple.TCC/TCC.db"
INPUT_SERVICE=kTCCServiceAccessibility

APP_ID=com.apple.Terminal
PERMISSION_GRANTED=1
APP_CSREQ=X'FADE0C000000003000000001000000060000000200000012636F6D2E6170706C652E5465726D696E616C000000000003'

### working, but no csreq
sudo sqlite3 "$DATABASE_SYSTEM" "REPLACE INTO access VALUES('"$INPUT_SERVICE"','"$APP_ID"',0,$PERMISSION_GRANTED,1,NULL,NULL,NULL,?,NULL,0,?);"

### working with csreq
sqlite3 "$DATABASE_USER" "REPLACE INTO access VALUES('"$INPUT_SERVICE"','"$APP_ID"',0,$PERMISSION_GRANTED,1,$APP_CSREQ,NULL,NULL,?,NULL,NULL,?);"

This still works, but macOS Mojave introduced some more security settings and for setting automation via command line correctly the csreqs of both apps, the automating, and the automated app, is needed, for example allowing Terminal to automate System Settings:

DATABASE_USER="/Users/"$USER"/Library/Application Support/com.apple.TCC/TCC.db"
SOURCE_APP_ID=com.apple.Terminal
SOURCE_APP_CSREQ=X'FADE0C000000003000000001000000060000000200000012636F6D2E6170706C652E5465726D696E616C000000000003'
PERMISSION_GRANTED=1
AUTOMATED_APP_ID=com.apple.systemevents
AUTOMATED_APP_CSREQ=X'FADE0C000000003400000001000000060000000200000016636F6D2E6170706C652E73797374656D6576656E7473000000000003'

sqlite3 "$DATABASE_USER" "REPLACE INTO access VALUES('kTCCServiceAppleEvents','"$SOURCE_APP_ID"',0,$PERMISSION_GRANTED,1,$SOURCE_APP_CSREQ,NULL,0,'"$AUTOMATED_APP_ID"',$AUTOMATED_APP_CSREQ,NULL,?);"

If the csreq is replaced with a questionmark "?", then the entry works, but does not show up in the GUI of the system preferences at all.

The System generates the csreq when you click allow when it asks for permissions and can be read from the database afterwards. To make correct entries via command line I would like to read / generate the correct csreq from the app on the command line without reading it from the tcc.db as it seems to change with every version of the app.

Thanks for any help in advance

Edit

A big thanks to Keith Johnson for the very good explanation and for solving the question for a big part. Following his answers I was able to implement the csreq in the new config file for my scripts.

https://github.com/tiiiecherle/osx_install_config/blob/master/_config_file/shellscriptsrc.sh

in the functions env_set_apps_security_permissions and env_set_apps_automation_permissions.

What I couldn't solve yet is getting the csreq for unsigned applets or droplets created with script editor or automator.

PATH_TO_APP="/Applications/brew_casks_update.app"
codesign --detached "$PATH_TO_APP".sig -s - "$PATH_TO_APP"
SOURCE_APP_CSREQ_STRING=$(codesign -d -r- --detached "$PATH_TO_APP".sig "$PATH_TO_APP")
echo "$SOURCE_APP_CSREQ_STRING" | csreq -r- -b /tmp/csreq.bin
Executable=/Applications/brew_casks_update.app/Contents/MacOS/applet error: invalid or corrupted code requirement(s) Requirement syntax error(s): line 2:1: unexpected end of file

If I add it manually and read the string from the database it has a valid csreq. The question would be fully solved if this could be figured out, too. That would be really nice.

Thanks

Answer 1

Background

The format of the csreq blob itself doesn't appear too complicated, and the source code found in the Security.Framework can assist in decoding the meaning if you want to get fancy. It's basically a magic header(the 0xFADE0C00) followed by a 32bit length(the size of the blob) and then some number of different "operations".

Thankfully there are already utilities we can use to manipulate the csreq blob, so we don't need to dig too deep into this.

Decoding the csreq blob, and finding where it comes from

Let me take your example above for Terminal.app. Apple ships a tool called csreq that can be used to convert the requirement from it's binary representation to a text representation(and back again).

# Convert the hex string into a binary blob
$ BLOB="FADE0C000000003000000001000000060000000200000012636F6D2E6170706C652E5465726D696E616C000000000003"
$ echo "$BLOB" | xxd -r -p > terminal-csreq.bin

# Ask csreq to tell us what it means
$ csreq -r- -t < terminal-csreq.bin
identifier "com.apple.Terminal" and anchor apple

So that csreq blob just says it wants to match the application with identifier "com.apple.Terminal" and anchor apple. We can inspect Terminal.app and find the same requirements string(under "designated")

$ codesign -d -r- /Applications/Utilities/Terminal.app
Executable=/Applications/Utilities/Terminal.app/Contents/MacOS/Terminal
designated => identifier "com.apple.Terminal" and anchor apple

Lets look at another example, this time for virtualbox from your repository:

# Convert the hex string into a binary blob
BLOB="FADE0C00000000AC0000000100000006000000020000001D6F72672E7669727475616C626F782E6170702E5669727475616C426F78000000000000060000000F000000060000000E000000010000000A2A864886F76364060206000000000000000000060000000E000000000000000A2A864886F7636406010D0000000000000000000B000000000000000A7375626A6563742E4F550000000000010000000A564235453254563936330000"
$ echo "$BLOB" | xxd -r -p > vbox-csreq.bin

# Ask csreq to tell us what it means
$ csreq -r- -t < vbox-csreq.bin
identifier "org.virtualbox.app.VirtualBox" and anchor apple generic and certificate 1[field.1.2.840.113635.100.6.2.6] /* exists */ and certificate leaf[field.1.2.840.113635.100.6.1.13] /* exists */ and certificate leaf[subject.OU] = VB5E2TV963

# ask codesign what the requirement text from the application itself is
$ codesign -d -r- /Applications/VirtualBox.app
Executable=/Applications/VirtualBox.app/Contents/MacOS/VirtualBox
designated => identifier "org.virtualbox.app.VirtualBox" and anchor apple generic and certificate 1[field.1.2.840.113635.100.6.2.6] /* exists */ and certificate leaf[field.1.2.840.113635.100.6.1.13] /* exists */ and certificate leaf[subject.OU] = VB5E2TV963

Again we can see that the designated field codesign returns is the same as what the csreq blob contains.

If you're curious about the actual meaning of the requirements string, apple has some documentation on that.

---

Building a new csreq blob

Now we know what the csreq blob contains and where to find equivalent information, we need to convert it to the binary format the tcc database requires. The csreq tool we used above for decoding the binary blob is also capable of converting the text representation into binary.

# Get the requirement string from codesign
$ REQ_STR=$(codesign -d -r- /Applications/Utilities/Terminal.app/ 2>&1 | awk -F ' => ' '/designated/{print $2}')

# Convert the requirements string into it's binary representation(sadly it seems csreq requires the output to be a file; so we just throw it in /tmp)
$ echo "$REQ_STR" | csreq -r- -b /tmp/csreq.bin

# Convert the binary form to hex, and print it nicely for use in sqlite
$ REQ_HEX=$(xxd -p /tmp/csreq.bin  | tr -d '\n')
$ echo "X'$REQ_HEX'"
X'fade0c000000003000000001000000060000000200000012636f6d2e6170706c652e5465726d696e616c000000000003'

This hex string is identical to what you had above from the tcc database.

---

Bonus: Unsigned binaries

If you want to trust a script/binary that is unsigned and try to use the above method, you'll run into some issues:

$ codesign -d -r- ./hello.sh
./hello.sh: code object is not signed at all

Reading the documentation about the Code Signing Requirement Language, there is a small note under Code Directory Hash(emphasis mine)

Because the code directory changes whenever the program changes in a nontrivial way, this test can be used to unambiguously identify one specific version of a program. When the operating system signs an otherwise unsigned program (so that the keychain or Parental Controls can recognize the program, for example), it uses this requirement.

So for unsigned programs, an ad-hoc signature is generated and used for identification for things like TCC(among others).

The codesign tool can be used to create an ad-hoc signature which we can get the requirement string from.

$ cat hello.sh 
#!/bin/bash
echo "Hello World"

# Create a detached signature(so as not to modify the original binary)
$ codesign --detached ./hello.sh.sig -s - ./hello.sh

# Display the designated identifier from the detached signature
$ codesign -d -r- --detached ./hello.sh.sig ./hello.sh 
Executable=/Users/keith/hello.sh
host => identifier "com.apple.bash" and anchor apple
# designated => cdhash H"70212a41efea9849e7a88afa946afa3e1b559cbe" or cdhash H"9044184bcced89d2f4bf1d75ec61a7537871eee7"

From here we can continue with the same process given above.

---