Jenkins Git Plugin with Git LFS fails with 'Bad Credentials' on Mac slave

Question

At my company we have been setting up a Jenkins CI system against our internal GitHub Enterprise instance using the Jenkins Git Plugin (2.4.2) and Git LFS (1.1.2). Despite having valid credentials stored in Jenkins and also stored on the Mac slave using the globally configured credentials, checking out the repository always ends with this failure:

Caused by: hudson.plugins.git.GitException: Command "git checkout -f ___my_commit_hash___ returned status code 128:
stdout:
stderr: Downloading ___my_lfs_file___ (754 B)
Error accessing media: ___my_lfs_file___ (___my_lfs_file_hash___)

Errors logged to .git\lfs\objects\logs\___my_lfs_log___.log
Use `git lfs logs last` to view the log.
error: external filter git-lfs smudge %f failed 2
error: external filter git-lfs smudge %f failed
fatal: ___my_lfs_file___: smudge filter lfs failed

And the git log shows:

Git credentials for https://git.mycompany.com/myrepository.git not found.

or:

Bad Credentials

When I log onto the slave and directly run the commands, everything works; only the Jenkins slave process triggers the errors. This also only seems to affect our Mac slave, the Windows slave uses the stored credentials and properly runs Git LFS. How can I configure the Git Plugin to properly run Git LFS on a Mac slave?

Answer 1

The issue with the Mac slave involved many facets.

First, we were launching the slave with Launch slave agents on Unix machines via SSH. This starts a non-interactive shell to launch the Jenkins slave.jar java process. The fact that this is a not a login shell means that it won't have access to the credentials stored in the keychain without some additional setup. It also means it won't pick up any paths from /etc/paths (used by Homebrew when installing git-lfs) without some bash profile configurations. If we had been launching Jenkins on the slave using Launch slave agents via Java Web Start (which is how our Windows slave was configured) it's possible that we may not have been affected by the access issues (but this wasn't tested).

We had Git LFS installed using Homebrew which modifies /etc/paths to add /usr/local/bin. The paths in this file are loaded by .profile which only gets used for login shells. To add /usr/local/bin to the path (so that git-lfs could be found by the Jenkins slave process) we added a ~/.bashrc file with the modified path.

To enable Git LFS, it was also necessary to modify ~/.gitconfig (using git config --global) with the following values:

[filter "lfs"]
    clean = git-lfs clean %f
    smudge = git-lfs smudge %f
    required = true

Without this, checkout would succeed but Git LFS would silently not run leaving its stubs in the repository.

It was also necessary to add a credentials helper to the global git configuration (see this link for more details).

[credential]
    helper = "osxkeychain "

The Git Plugin uses the slightly weird approach of init+fetch rather than clone to download the repository (see JENKINS-30318 for more details). This was added to work around a credentials issue (which should no longer be necessary). The plugin uses the local git config file (which is why init+fetch was necessary) to temporarily store credentials which are subsequently removed. Specific commands are wrapped with this credentials setup (fetch is one). Unfortunately, the git checkout command was not wrapped with the credentials. This means that when git-lfs is invoked, it must get its credentials from somewhere other than what is stored on Jenkins. From the Git LFS API, it shows that Git LFS can use gitcredentials to access the server (GitHub). On our Windows slave, this just worked. On the Mac slave however, because the Jenkins process was not run in a login shell, Git LFS didn't have permission to access the user keychain and would fail. Opening Keychain Access and moving the keys from the 'Login' category to the 'System' category (see this comment) and allowing all applications to access the key (see this answer), we were finally able to get checkout with Git LFS working.