SSH Public Key Authentication with Libssh2 C++

Question

I am working on a project where I am doing port forwarding to MySQL using libssh2 in C++. I've got it working for username/password authentication but I now want do it using public/private key authentication. The documentation for libssh2 is pretty poor so I am having difficult working out what I need to do.

What I am trying to do is have an Android app post data to my C++ app, where C++ will be given the SSH details along with the auth key and it creates the SSH tunnel. C++ is getting the key fine and then I am doing the following to do the public key authentication.

else if (this->getAuthMethod() == SupportedAuthMethods::AUTH_PUBLICKEY)
    {

        string test = this->getSSHPrivateKey();

        boost::replace_all(test, "\n", "");

        unsigned char * key = (unsigned char *)test.c_str();

        size_t sizeofkey = strlen((char*)key);
        cout << key << endl;
        stringstream logstream;
        logstream << "Using public key authentication for SSH Host: " << this->getSSHHostnameOrIPAddress();
        this->bitsLibrary->writeToLog(logstream.str(), "SSHTunnelForwarder", "authenticateSSHServer");
        if (chosenAuthMethod & SupportedAuthMethods::AUTH_PUBLICKEY)
        {
            //int result = 0;
            int result = libssh2_userauth_publickey(this->session, this->getUsername().c_str(), key, sizeofkey, SSHTunnelForwarder::publicKeyAuthComplete, 0);
            if (result != 0)
            {
                char * error = NULL;
                int len = 0;
                int errbuf = 0;
                libssh2_session_last_error(this->session, &error, &len, errbuf);
                this->bitsLibrary->writeToLog(std::string(error), "SSHTunnelForwarder", "auth");
                JSONResponseGenerator jsonResponse;
                jsonResponse.generateJSONResponse(API_AUTH_FAILURE, "InvalidPublicKey");
                return jsonResponse.getJSONString();

            }
        }
    }

I read somewhere that it shouldn't have new lines, that's why I'm replacing the \n for blank characters but with or without this it doesn't make a difference.

The very basic documentation mentions that one of the arguments for libssh2_userauth_publickey is a callback as follows:

int name(LIBSSH2_SESSION *session, unsigned char **sig, size_t *sig_len, const unsigned char *data, size_t data_len, void **abstract);

but I cannot find any information anywhere as to what this callback is or what it should contain. In my function call to libssh2_userauth_publickey I pass in SSHTunnelForwarder::publicKeyAuthComplete and currently this function just has the following:

int SSHTunnelForwarder::publicKeyAuthComplete(LIBSSH2_SESSION *session, unsigned char **sig, size_t *sig_len,
    const unsigned char *data, size_t data_len, void **abstract)
{
    cout << "In SSH Auth Callback" << endl;
    return 0;
}

This method above doesn't get called, although I'm expecting that this isn't even close to being correct.

When I run my code above the result of libssh2_userauth_publickey is 19 and the error that is returned from the get_last_message() method is invalid public key. I know the public key file is fine as I can use it in an SSH app on Android and can successfully authenticate with my server.

UPDATE 1

I've managed to make some progress, I found there is a function called:

libssh2_userauth_publickey_frommemory(LIBSSH2_SESSION *session, const char *username, size_t username_len, const char *publickeyfiledata, size_t publickeyfiledata_len, const char *privatekeyfiledata, size_t privatekeyfiledata_len, const char *passphrase);

The libssh2 version that I had was from NuGet Package manager and found that the latest version in this is actually quite old, so I've build the latest 1.7.0 version from GitHub inside Visual Studio and relinked my project to this new version.

I have now replaced the original publickey auth function I was using with this version so my code now looks like this instead:

int result = libssh2_userauth_publickey_frommemory(this->session, username.c_str(), username.length(), nullptr, 0, test.c_str(), sizeofkey, nullptr);

This function confuses me a little that it wants the public key and the private key, for everything I know about SSH (admittedly not a huge amount) the public key stays on the server, and the user wanting to login only has access to the private key.

I found that someone else was asking this and a patch was done to allow a null pointer for the public key, so I have passed nullptr for the public key and 0 for the public key length. I've also put a nullptr for the certificate passphrase as this hasn't been set.

When I run this code now I now get the error -18 which is invalid username/public key combination. Again though, I know the username and private key file I am using is correct as I can use it in an SSH client application on Android to connect to my SSH server.

Answer 1

I've found the solution. What I did in update 1 was the actual fix but my server key got changed for some reason not sure why, so when it said that it couldn't link the username with the key it was correct.

The complete fix is as follows:

Don't use the libssh2 from NuGet its very old. I downloaded the latest source from GitHub at https://github.com/libssh2/libssh2/releases and then completed the follwing steps to build libssh2 within Visual Studio.

• Shove all the .c files in libssh2/src into an empty Win32 C++ (DLL or Static Library) project except libgcrypt.c/openssl.c of which you only pick the one appropriate to your crypto library.
• Add your OpenSSL or libgcrypt include directory to the project include path
• Add libssh2/include to the project include path
• Add libssh2/win32 to the project include path
• Add the appropriate crypto libraries to the project Additonal Libraries list
• Build
• Job done

Steps above taken from https://www.libssh2.org/mail/libssh2-devel-archive-2012-09/0029.shtml

The libssh2 was linked to openssl during compilation.

Link your project to the latest build you've just done for libssh2 and use the method

libssh2_userauth_publickey_frommemory(LIBSSH2_SESSION *session, const char *username, size_t username_len, const char *publickeyfiledata, size_t publickeyfiledata_len, const char *privatekeyfiledata, size_t privatekeyfiledata_len, const char *passphrase);

And that's it.