Capistrano Deploy Failing on git:check - Permission denied (publickey)

Question

Local OS: Windows 10 Pro (using Git Bash as my terminal)

Staging Server OS: Ubuntu 16.04 LTS

I've been struggling to get a basic Capistrano deploy working on my staging server. I followed this guide to set up Capistrano.

The deployment process always fails on the git:check stage due to allegedly not having permission to access my repo on GitLab. I am sure my SSH Agent forwarding is working because I am able to SSH into my server and access GitLab with my SSH key. The SSH key is not stored anywhere on my server:

$ ssh [email protected]

deploy@MyServer:~$ ssh -T [email protected]
debug1: client_input_channel_open: ctype [email protected] rchan 2 win 65536 max 16384
debug1: channel 1: new [authentication agent connection]
debug1: confirm [email protected]
Welcome to GitLab, Alexander!debug1: channel 1: FORCE input drain

None of the questions about this issue have been working for me so far.

Here are my deploy files:

deploy.rb

set :application, "myapp"
set :branch, "master"
set :repo_url, "[email protected]:MyApp/myapp.git"

# Defaults to false
# Skip migration if files in db/migrate were not modified
set :conditionally_migrate, true

set :ssh_options, { forward_agent: true }

set :rvm_ruby_version, '2.2.6'

# Default deploy_to directory is /var/www/my_app_name
set :deploy_to, "/var/www/#{fetch(:application)}/"
set :deploy_user, "deploy"

# Tells Capistrano to store config/database.yml file inside a directory called /shared, which is meant for any files
# we want to persist between deploys
set :linked_files, fetch(:linked_files, []).push('config/database.yml', 'config/secrets.yml')

# Directories that are meant to persist between deploys, and they will also be stored inside /shared
set :linked_dirs, fetch(:linked_dirs, []).push('bin', 'log', 'tmp/pids', 'tmp/cache', 'tmp/sockets', 'vendor/bundle', 'public/system', 'public/uploads')

# The specs that should be run before deployment is allowed to continue
set :tests, []

# Delayed Job Config: https://github.com/AgileConsultingLLC/capistrano3-delayed-job
set :delayed_job_workers, 3

# Keep the last 5 deploys for rollback purposes
set :keep_releases, 5

namespace :deploy do
  after :restart, :clear_cache do
    on roles(:web), in: :groups, limit: 3, wait: 10 do
      # Here we can do anything such as:
      # within release_path do
      #   execute :rake, 'cache:clear'
      # end
    end
  end
end

staging.rb

set :stage, :staging
set :rails_env, :staging

role :app, %w{[email protected]}
role :web, %w{[email protected]}
role :db,  %w{[email protected]}

And here is the Capistrano debug log when I deploy:

$ bundle exec cap staging deploy --trace
** Invoke staging (first_time)
** Execute staging
** Invoke load:defaults (first_time)
** Execute load:defaults
** Invoke rvm:hook (first_time)
** Invoke passenger:rvm:hook (first_time)
** Invoke passenger:test_which_passenger (first_time)
** Execute passenger:test_which_passenger
** Execute passenger:rvm:hook
** Execute rvm:hook
** Invoke rvm:check (first_time)
** Execute rvm:check
rvm 1.29.3 (latest) by Michal Papis, Piotr Kuczynski, Wayne E. Seguin [https://rvm.io]
ruby-2.2.6
ruby 2.2.6p396 (2016-11-15 revision 56800) [x86_64-linux]
** Invoke bundler:map_bins (first_time)
** Invoke passenger:bundler:hook (first_time)
** Execute passenger:bundler:hook
** Execute bundler:map_bins
** Invoke deploy:set_rails_env (first_time)
** Execute deploy:set_rails_env
** Invoke deploy:set_linked_dirs (first_time)
** Execute deploy:set_linked_dirs
** Invoke deploy:set_rails_env
** Invoke deploy (first_time)
** Execute deploy
** Invoke deploy:starting (first_time)
** Execute deploy:starting
** Invoke deploy:check (first_time)
** Invoke git:check (first_time)
** Invoke git:wrapper (first_time)
** Execute git:wrapper
00:00 git:wrapper
      01 mkdir -p /tmp
    ✔ 01 [email protected] 0.286s
      Uploading /tmp/git-ssh-myapp-staging-localuser.sh 100.0%
      02 chmod 700 /tmp/git-ssh-myapp-staging-localuser.sh
    ✔ 02 [email protected] 0.277s
** Execute git:check
00:01 git:check
      01 git ls-remote [email protected]:MyApp/myapp.git HEAD
      01 Permission denied (publickey).
      01 fatal: Could not read from remote repository.
      01
      01 Please make sure you have the correct access rights
      01 and the repository exists.
cap aborted!
SSHKit::Runner::ExecuteError: Exception while executing as [email protected]: git exit status: 128
git stdout: Nothing written
git stderr: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
C:/Ruby22/lib/ruby/gems/2.2.0/gems/sshkit-1.15.1/lib/sshkit/runners/parallel.rb:15:in `rescue in block (2 levels) in execute'
C:/Ruby22/lib/ruby/gems/2.2.0/gems/sshkit-1.15.1/lib/sshkit/runners/parallel.rb:11:in `block (2 levels) in execute'

Caused by:
SSHKit::Command::Failed: git exit status: 128
git stdout: Nothing written
git stderr: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
C:/Ruby22/lib/ruby/gems/2.2.0/gems/sshkit-1.15.1/lib/sshkit/command.rb:99:in `exit_status='
C:/Ruby22/lib/ruby/gems/2.2.0/gems/sshkit-1.15.1/lib/sshkit/backends/netssh.rb:169:in `execute_command'
C:/Ruby22/lib/ruby/gems/2.2.0/gems/sshkit-1.15.1/lib/sshkit/backends/abstract.rb:141:in `block in create_command_and_execute'
C:/Ruby22/lib/ruby/gems/2.2.0/gems/sshkit-1.15.1/lib/sshkit/backends/abstract.rb:141:in `tap'
C:/Ruby22/lib/ruby/gems/2.2.0/gems/sshkit-1.15.1/lib/sshkit/backends/abstract.rb:141:in `create_command_and_execute'
C:/Ruby22/lib/ruby/gems/2.2.0/gems/sshkit-1.15.1/lib/sshkit/backends/abstract.rb:74:in `execute'
C:/Ruby22/lib/ruby/gems/2.2.0/gems/capistrano-3.10.1/lib/capistrano/scm/git.rb:77:in `git'
C:/Ruby22/lib/ruby/gems/2.2.0/gems/capistrano-3.10.1/lib/capistrano/scm/git.rb:38:in `check_repo_is_reachable'
C:/Ruby22/lib/ruby/gems/2.2.0/gems/capistrano-3.10.1/lib/capistrano/scm/tasks/git.rake:19:in `block (4 levels) in eval_rakefile'
C:/Ruby22/lib/ruby/gems/2.2.0/gems/sshkit-1.15.1/lib/sshkit/backends/abstract.rb:93:in `with'
C:/Ruby22/lib/ruby/gems/2.2.0/gems/capistrano-3.10.1/lib/capistrano/scm/tasks/git.rake:18:in `block (3 levels) in eval_rakefile'
C:/Ruby22/lib/ruby/gems/2.2.0/gems/sshkit-1.15.1/lib/sshkit/backends/abstract.rb:29:in `instance_exec'
C:/Ruby22/lib/ruby/gems/2.2.0/gems/sshkit-1.15.1/lib/sshkit/backends/abstract.rb:29:in `run'
C:/Ruby22/lib/ruby/gems/2.2.0/gems/sshkit-1.15.1/lib/sshkit/runners/parallel.rb:12:in `block (2 levels) in execute'
Tasks: TOP => deploy:check => git:check
The deploy has failed with an error: Exception while executing as [email protected]: git exit status: 128
git stdout: Nothing written
git stderr: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
** Invoke deploy:failed (first_time)
** Execute deploy:failed


** DEPLOY FAILED
** Refer to log/capistrano.log for details. Here are the last 20 lines:


 DEBUG [6b6ba2d0] Finished in 0.471 seconds with exit status 0 (successful).

 DEBUG [c6e2d7dc] Running ~/.rvm/bin/rvm 2.2.6 do ruby --version as [email protected]

 DEBUG [c6e2d7dc] Command: ~/.rvm/bin/rvm 2.2.6 do ruby --version

 DEBUG [c6e2d7dc]       ruby 2.2.6p396 (2016-11-15 revision 56800) [x86_64-linux]

 DEBUG [c6e2d7dc] Finished in 0.608 seconds with exit status 0 (successful).

  INFO [fd5500a8] Running /usr/bin/env mkdir -p /tmp as [email protected]

 DEBUG [fd5500a8] Command: /usr/bin/env mkdir -p /tmp

  INFO [fd5500a8] Finished in 0.286 seconds with exit status 0 (successful).

 DEBUG Uploading /tmp/git-ssh-myapp-staging-localuser.sh 0.0%

  INFO Uploading /tmp/git-ssh-myapp-staging-localuser.sh 100.0%

  INFO [f33d4873] Running /usr/bin/env chmod 700 /tmp/git-ssh-myapp-staging-localuser.sh as [email protected]

 DEBUG [f33d4873] Command: /usr/bin/env chmod 700 /tmp/git-ssh-myapp-staging-localuser.sh

  INFO [f33d4873] Finished in 0.277 seconds with exit status 0 (successful).

  INFO [86d3cd5a] Running /usr/bin/env git ls-remote [email protected]:MyApp/myapp.git HEAD as [email protected]

 DEBUG [86d3cd5a] Command: ( export GIT_ASKPASS="/bin/echo" GIT_SSH="/tmp/git-ssh-myapp-staging-localuser.sh" ; /usr/bin/env git ls-remote [email protected]:MyApp/myapp.git HEAD )

 DEBUG [86d3cd5a]       Permission denied (publickey).

 DEBUG [86d3cd5a]       fatal: Could not read from remote repository.



Please make sure you have the correct access rights

and the repository exists.

I've tried manually running the commands that Capistrano attempts both locally and on my remote server, and they all succeed. I'm really stuck, and any help would be hugely appreciated!

Answer 1

Add your public key to the list of deploy keys in your repository setting by following the below steps:

Local machine setup:

1. Check if your local system has ~/.ssh/id_rsa.pub key file. If not, create a new one:

   $ ssh-keygen -t rsa
   

2. Add the newly created public key ~/.ssh/id_rsa.pub to the repository's deployment(access) keys in settings:

   • Bitbucket :
     • https://confluence.atlassian.com/bitbucket/set-up-ssh-for-git-728138079.html#SetupSSHforGit-Step4.InstallthepublickeyonyourBitbucketaccount
   • Github :
     • https://developer.github.com/v3/guides/managing-deploy-keys/#deploy-keys
   • GitLab :
     • https://docs.gitlab.com/ce/ssh/README.html#deploy-keys

3. Load the keys to ssh-agent:

   Check whether ssh-agent is running, If not, start the ssh agent

   $ ssh-agent /bin/bash
   

   Add the id_rsa key to the agent:

   $ ssh-add ~/.ssh/id_rsa
   

   Note: Sometimes, this step(Step - 3) needs to be done before each deployment if you receive "Access denied to the repository" error during deployment.

4. Add your local SSH Key to deployment server Authorized Keys file (remember to replace the port number with your customized port number):

   $ cat ~/.ssh/id_rsa.pub | ssh -p port_num user@server_ip 'cat >> ~/.ssh/authorized_keys'
   

Ref: https://www.digitalocean.com/community/tutorials/deploying-a-rails-app-on-ubuntu-14-04-with-capistrano-nginx-and-puma

deploy.rb

Try to change the ssh_options in deploy.rb file as below:

set :ssh_options, { forward_agent: true, user: "deploy", auth_methods: ['publickey'], keys: %w(~/.ssh/privatekey.pem) }

Replace ~/.ssh/privatekey.pem with the path to your SSH private key file path.

Deploy:

Just run cap staging deploy to deploy to the server.

Replace staging in cap deploy command with the environment as needed.

Answer 2

Thanks to everyone who answered, I've managed to find a solution! The main culprit was Git Bash, which, for whatever reason, was not changing the permissions on my ~/.ssh directory to 0700 when I ran chmod 700 ~/.ssh. This prevented SSH agent forwarding from working when Capistrano was deploying but not when I manually SSH'd into my server. I decided to try using Bash on Ubuntu on Windows (BUW) instead of Git Bash, and sure enough, my deploy worked! I copied over the same exact configuration and keys from Git Bash over to BUW. The only difference is that I was able to change the permissions on BUW's ~/.ssh directory to 0700. With that said, here is the solution to my problem:

1. Create a deploy key and add it to GitLab

As @Onur and @grizzthedj, and @Gokul M indicated, I needed to create a deploy key for GitLab and authorize it on my server. Here's how I did that:

1. Generate a new SSH key on my local machine: ssh-keygen -t rsa -b 4096
2. Copy the output of the public key: cat ~/.ssh/id_rsa.pub
3. SSH into my server
4. Add the public key to the end of ~/.ssh/authorized_keys
5. Back on my local machine, open up a browser, log into GitLab, go to my repository page, and paste the public key in Settings > Repository > Deploy Keys

2. Use BUW instead of Git Bash

1. On my local machine, I set up BUW to start the SSH agent on session load using the instructions from this SO answer.
2. Change the permissions on ~/.ssh: chmod 700 ~/.ssh

3. Start the SSH agent and add my deploy key to it:

   eval $(ssh-agent -s)
   ssh-add ~/.ssh/id_rsa
   

I've removed the set :ssh_options line from my deploy.rb file because it works just fine without it.

And that's it! It looks like I'll be deploying using BUW from now on.

Answer 3

Based on your cap deploy output, it looks like SSH connectivity from your laptop to your server is not the issue.

00:00 git:wrapper
      01 mkdir -p /tmp
    ✔ 01 [email protected] 0.286s
      Uploading /tmp/git-ssh-myapp-staging-localuser.sh 100.0%
      02 chmod 700 /tmp/git-ssh-myapp-staging-localuser.sh
    ✔ 02 [email protected] 0.277s 

This issue is when capistrano tries to run git ls-remote [email protected]:MyApp/myapp.git HEAD over SSH.

You need to add the public key to your SSH Keys in GitLab, since SSH agent forwarding requires installation of public keys on all target systems. Gitlab, in your case, is a target system.

cat ~/.ssh/id_rsa.pub    # Copy the contents of your public key(filename may be different)

Login to GitLab, and paste the public key into your repo's SSH Keys (found in repo settings) and you should be good.

You also need to specify the user that you created the SSH key with in ssh_options.

set :ssh_options, {
    forward_agent: true,
    user: 'deploy',
}