Jsoup http logging

Question

Is there a way to log the http request and response? Let's assume the below request

Connection.Response res = Jsoup.connect("LOGIN_URL_HERE")
            .data("user", "USER", "pass", "PASS")
            .method(Connection.Method.POST)
            .execute();

How can I log the http request and response? Please mind that I want the HTTP and not just the HTML that will be parsed.

Answer 1

By default jsoup uses a implementation of java.net.HttpURLConnection So I suppose you need to turn on logging for that implementation (probably: sun.net.www.protocol.http.HttpURLConnection) or for java.net.

There is a system property that will enable logging for java net utils

-Djavax.net.debug=all

Answer 2

As Jsoup lacks logging (version I'm using: 1.12.1) and using the -Djavax.net.debug=all JVM argument logs are too verbose, the best way I found is to decorate the HttpConnection class, so one can customize what is logged. To achive this, the execute method call needs to be surrounded by logging the properties of the Connection.Request and Connection.Response.

Sample implementation using SLF4J:

import org.jsoup.Connection;
import org.jsoup.helper.HttpConnection;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.io.IOException;

public class DiagnosticConnection extends HttpConnection {
    static final Logger LOG = LoggerFactory.getLogger(DiagnosticConnection.class);

    @Override
    public Connection.Response execute() throws IOException {
        log(this.request());
        Connection.Response response = super.execute();
        log(response);

        return response;
    }

    public static Connection connect(String url) {
        Connection connection = new DiagnosticConnection();
        connection.url(url);
        return connection;
    }

    private static void log(Connection.Request request) {
        LOG.info("========================================");
        LOG.info("[url] {}", request.url());
        LOG.info("== REQUEST ==");
        logBase(request);
        LOG.info("[method] {}", request.method());
        LOG.info("[data] {}", request.data());
        LOG.info("[request body] {}", request.requestBody());
    }

    private static void log(Connection.Response response) {
        LOG.info("== RESPONSE ==");
        logBase(response);
        LOG.info("[code] {}", response.statusCode());
        LOG.info("[status msg] {}", response.statusMessage());
        LOG.info("[body] {}", response.body());
        LOG.info("========================================");
    }

    private static void logBase(Connection.Base<?> base) {
        LOG.info("[headers] {}", base.headers());
        LOG.info("[cookies] {}", base.cookies());
    }

}

When using the decorator, instead of Jsoup.connect(<URL>) you should use DiagnosticConnection.connect(<URL>)