What is the correct way to configure an SCP route with publickey authentication in Camel?

Question

I'm currently leveraging Apache Camel (version 2.20.2 at the time of writing) as part of a larger ETL flow to copy processed files from the Camel box to another machine.

However, I'm having a devil of a time dealing with the SCP configuration. The intention is to make it so that I don't have to provide much outside of where the private key lives and where the known hosts live.

Below is a sample route. The sample route is only for the sake of conversation; it may not be accurate, but the intention here is not to display the upstream portion as "working". I am assured that the file generation piece is working due to tests I have written upstream for it.

---

What works:

If I specify my username and password and disable strict host key checking, my route works.

from("direct:init")
    .to("file:///tmp")
    .to("scp://my.server.local?username=makoto&password=XXXXXX" + 
                              "&preferredAuthentications=password" +
                              "&strictHostKeyChecking=no");

Of course, not having strict host checking is an absolute non-starter due to policy.

What doesn't work:

• If I specify my username and password and do not disable strict host key checking, I get this error:

  com.jcraft.jsch.JSchException: reject HostKey: my.server.local
      at com.jcraft.jsch.Session.checkHost(Session.java:789) ~[jsch-0.1.54.jar:na]
      at com.jcraft.jsch.Session.connect(Session.java:345) ~[jsch-0.1.54.jar:na]
      at org.apache.camel.component.scp.ScpOperations.createSession(ScpOperations.java:284) [camel-jsch-2.20.2.jar:2.20.2]
      at org.apache.camel.component.scp.ScpOperations.connect(ScpOperations.java:179) [camel-jsch-2.20.2.jar:2.20.2]
  

• If I specify my username and password, do not disable strict host key checking and do not specify my preferred authentication type as "password", I get the same error as above.

• If I omit my password in favor of specifying the path to my private key, and I disable strict host key checking, I get this error:

  com.jcraft.jsch.JSchException: Auth cancel
      at com.jcraft.jsch.Session.connect(Session.java:518) ~[jsch-0.1.54.jar:na]
      at org.apache.camel.component.scp.ScpOperations.createSession(ScpOperations.java:284) [camel-jsch-2.20.2.jar:2.20.2]
  

• If I do all of the above and include publickey as my preferred authentication, I get this error:

  com.jcraft.jsch.JSchException: Auth fail
      at com.jcraft.jsch.Session.connect(Session.java:519) ~[jsch-0.1.54.jar:na]
      at org.apache.camel.component.scp.ScpOperations.createSession(ScpOperations.java:284) [camel-jsch-2.20.2.jar:2.20.2]
      at org.apache.camel.component.scp.ScpOperations.connect(ScpOperations.java:179) [camel-jsch-2.20.2.jar:2.20.2]
  

  In this scenario it seems Camel is outright ignoring my user, and is electing to use its own:

  2018-02-19 10:46:15.142 DEBUG 23940 --- [obfuscated-route] o.a.camel.component.scp.ScpOperations    : Passphrase for camel-jsch
  2018-02-19 10:46:15.142  WARN 23940 --- [obfuscated-route] o.a.camel.component.scp.ScpOperations    : Private Key authentication not supported
  2018-02-19 10:46:15.142 DEBUG 23940 --- [obfuscated-route] o.a.camel.component.scp.ScpOperations    : Passphrase for camel-jsch
  2018-02-19 10:46:15.142  WARN 23940 --- [obfuscated-route] o.a.camel.component.scp.ScpOperations    : Private Key authentication not supported
  

---

Needless to say, the error messages here don't give me much of anything to go off of, since:

• They wildly vary given the circumstance
• They are configuration specific; it's as if there's something missing with each piece of configuration

---

With this in mind, what is the correct way to configure this? Scattered documentation on the Camel mailing list doesn't produce anything concrete, and the SCP documentation is of little actual help for these circumstances.

Some other environmental things to note:

• I do not "control" either the machine that Camel will live on, nor the machine that will be receiving the files.
• I am testing this locally on my machine and my public key is absolutely contained on the host machine's authorized_keys file.
• The known_hosts entry on my machine is hashed/obfuscated.

Answer 1

This feels incredibly backwards and wrong, but I believe I've solved my own problem.

The executive summary:

• JSch is looking for a match on your hashed hostname.
• If it cannot find a match through all of your identities on your machine, it will mark the repository as "not included".
• If the repository is not marked as "OK" and strict host key checking is enabled, it will register this as a failure.

The executive solution:

• The host key must be added to your system (through the use of ssh-keyscan -t rsa -H <hostname>. Simply having the host key on the target machine is insufficient.
• If your private key has a passphrase, you are still obligated to provide it.

---

Interestingly enough the fact that Camel was seemingly using its own user and identity to connect to the server was a bit of a smoking gun. This led me to observe the code flow from the point of the exception.

The first thing I observed was that the hashed host was nowhere to be found in the identities vector that JSch was iterating over. This meant, in spite of the fact that I've SSHed to this machine prior, JSch was having none of it and was oblivious to its existence.

So I found some inspiration from this Server Fault question, given that I needed to add a new, hashed host to my own identities file.

Effectively, I utilized this answer since it seems to be the most practical and safest approach:

• ssh-keyscan -t rsa -H my.server.local
• Explicitly copy the RSA -only hash from the output of that command (there can be up to 3 lines, I believe) and append it to the end of my known_hosts file

I did discover, however, that authentication was still inexplicably failing. In an effort to debug and diagnose what was going on, I elected to keep the path to the private key, and add the private key's password to the system so that JSch could utilize it.

I could concede that the password being needed was an oversight by me, but the fact that it couldn't pick a sane private key to use caught me by surprise, especially given that normal SSH will iterate through them all.

Completed now, the full route looks something like this:

from("direct:init")
    .to("file:///tmp")
    .to("scp://my.server.local?username=makoto&privateKeyFilePassphrase=XXXXXX" + 
                          "&preferredAuthentications=publickey" +
                          "&privateKeyFile=/path/to/.ssh/id_rsa");