What should a default environment be for client-side (X509) Certification

Question

I have been building web downloaders over the years (e.g., using Apache HTTPClient and recently JBrowser [1]). These have worked OK till recently when some sites result in certification errors. I do not understand the details, and I cannot find a simple tutorial for people who know relatively little about certificates (e.g., what one looks like and how it obtained or created). This is a request for a default explanation of the simplest case and how to fix it. Typical error:

[2020-02-17T09:38:24.249][Instance 1][Port 57129] Warning: Single GUI Threadiong is enabled, FPS should be slower
[2020-02-17T09:38:29.737][Instance 1][Port 57129] Feb 17, 2020 9:38:29 AM com.sun.webkit.network.URLLoader doRun
[2020-02-17T09:38:29.737][Instance 1][Port 57129] WARNING: Unexpected error
[2020-02-17T09:38:29.737][Instance 1][Port 57129] java.io.IOException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target: https://osf.io/search/?q=coronavirus
[2020-02-17T09:38:29.737][Instance 1][Port 57129]   at com.machinepublishers.jbrowserdriver.StreamConnection.exec(StreamConnection.java:369)
[2020-02-17T09:38:29.737][Instance 1][Port 57129]   at com.machinepublishers.jbrowserdriver.StreamConnection.getResponseCode(StreamConnection.java:449)
[2020-02-17T09:38:29.737][Instance 1][Port 57129]   at com.sun.webkit.network.URLLoader.receiveResponse(URLLoader.java:414)
...

I can access the URL through browsers (Firefox, Chrome) and get HTML which represents what I want, but cannot access this programmatically.

I have read several accounts of how to fix this (e.g. [2]), but they generally refer to "your Keystore" or "trust manager" as if everyone knows what these are. I am concerned that if I don't know what I am doing, I could break security. I don't know how I add sites to these or whether I even should.

I am on MACOSX and appear to have a binary file

"/Library/Java/JavaVirtualMachines/jdk1.8.0_60.jdk/Contents/Home/JRE/lib/security/cacerts"

Some of the answers suggest I should have a file called "truststore.jks" but don't say where this should be or how it was created.

So I am asking for a simple explanation of the system components and the simplest way to fix it. In some cases, I can avoid it (e.g., by using curl from the command-line), so I don't know how much this is a Java (8) problem.

EDIT: Trust Store vs Key Store - creating with keytool seems to explain the difference between KeyStore and TrustStore, but I still don't have insight into what to do.

[1] http://machinepublishers.github.io/jBrowserDriver/com/machinepublishers/jbrowserdriver /JBrowserDriver.html [2] Using a custom truststore in java as well as the default one

Answer 1

Sorry, I'm not sure how much detail you need. Hope it will get you started. I can expand on the steps as you need. Just let me know.

Keystore stores private key and public cert and truststore store trusted certificates for other services you want to trust.Java bundles the truststore cacerts and it contains default, well known trusted certificate authorities.

As part of the ssl handshake the service presents its public certificate from its keystore to the client and client on the other hand verifies the presented certificate against its truststore.

For any programmatic access I would create a separate truststore file and import the certificates for services you trust. So in your case you have to import the certificate that is presented by https://osf.io/search/?q=coronavirus.

Once you have created the truststore you can reference truststore using jvm args.

Steps

1. Use KeyTool to create empty truststore
2. Use InstallCert utility or from browser to import the cert into truststore
3. Reference the newly created truststore in the jvm arg (javax.net.ssl.truststore)