Java 11 `HttpClient` downloading but not? (negative content-length)

Question

This works perfectly for some URLs like https://speed.hetzner.de/10GB.bin it prints a lot of the debug messages in my Subscriber so my console looks something like this:

I got 16384 bytes (6396)
Got data!
I got 16384 bytes (6397)
Got data!
I got 16384 bytes (6398)
Got data!
I got 16384 bytes (6399)

and taskmanager says that java is downloading with ~50Mbit/s (my full download speed)

However with the url you get for the file you can download here https://map.usbcraft.net/file/usbpc-downloads/java_http_client.fail this is very diffrent. The console only shows these messages:

Starting download...
Version: HTTP_1_1 Status code 200 Got headers back java.net.http.HttpHeaders@d4ccf412 { {accept-ranges=[bytes], cache-control=[max-age=0, no-cache, no-store], cf-ray=[46ed5b5aeaf9721d-AMS], connection=[keep-alive], content-length=[2147483648], content-type=[application/octet-stream], date=[Wed, 24 Oct 2018 15:08:07 GMT], expect-ct=[max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"], server=[cloudflare], set-cookie=[__cfduid=dcba8bb98269e665c521493baf25d0b6d1540393686; expires=Thu, 24-Oct-19 15:08:06 GMT; path=/; domain=.usbcraft.net; HttpOnly], x-bz-content-sha1=[none], x-bz-file-id=[4_zd729cdbbb41cd50850150a12_f201d3478bea24efd_d20181024_m130349_c001_v0001109_t0059], x-bz-file-name=[java_http_client.fail], x-bz-info-src_last_modified_millis=[1540386142160], x-bz-upload-timestamp=[1540386229000]} }
Content-Length parsed to int: -2147483648

and MySubscriber::onNext never gets called. However taskmanager is still showing my file to be downloading with full speed

Edit: After letting it run for quite a while it prints something more (this was for another URL but it should be the same for the updated url I'm currently letting it run):

I got an error fixed content-length: -298366259, bytes received: 0
Exception in thread "main" java.io.IOException: fixed content-length: -298366259, bytes received: 0
    at java.net.http/jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:565)
    at java.net.http/jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:119)
    at TestKt.main(Test.kt:29)
Caused by: java.io.IOException: fixed content-length: -298366259, bytes received: 0
    at java.net.http/jdk.internal.net.http.common.Utils.wrapWithExtraDetail(Utils.java:296)
    at java.net.http/jdk.internal.net.http.Http1Response$BodyReader.onReadError(Http1Response.java:742)
    at java.net.http/jdk.internal.net.http.Http1AsyncReceiver.checkForErrors(Http1AsyncReceiver.java:297)
    at java.net.http/jdk.internal.net.http.Http1AsyncReceiver.flush(Http1AsyncReceiver.java:263)
    at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SynchronizedRestartableTask.run(SequentialScheduler.java:175)
    at java.net.http/jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:147)
    at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:198)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.io.EOFException: EOF reached while reading
    at java.net.http/jdk.internal.net.http.Http1AsyncReceiver$Http1TubeSubscriber.onComplete(Http1AsyncReceiver.java:591)
    at java.net.http/jdk.internal.net.http.common.SSLTube$DelegateWrapper.onComplete(SSLTube.java:268)
    at java.net.http/jdk.internal.net.http.common.SSLTube$SSLSubscriberWrapper.complete(SSLTube.java:411)
    at java.net.http/jdk.internal.net.http.common.SSLTube$SSLSubscriberWrapper.onComplete(SSLTube.java:540)
    at java.net.http/jdk.internal.net.http.common.SubscriberWrapper.checkCompletion(SubscriberWrapper.java:443)
    at java.net.http/jdk.internal.net.http.common.SubscriberWrapper$DownstreamPusher.run1(SubscriberWrapper.java:322)
    at java.net.http/jdk.internal.net.http.common.SubscriberWrapper$DownstreamPusher.run(SubscriberWrapper.java:261)
    at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SynchronizedRestartableTask.run(SequentialScheduler.java:175)
    at java.net.http/jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:147)
    at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:198)
    at java.net.http/jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:271)
    at java.net.http/jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:224)
    at java.net.http/jdk.internal.net.http.common.SubscriberWrapper.onComplete(SubscriberWrapper.java:419)
    at java.net.http/jdk.internal.net.http.SocketTube$InternalReadPublisher$ReadSubscription.signalCompletion(SocketTube.java:632)
    at java.net.http/jdk.internal.net.http.SocketTube$InternalReadPublisher$InternalReadSubscription.read(SocketTube.java:833)
    at java.net.http/jdk.internal.net.http.SocketTube$SocketFlowTask.run(SocketTube.java:175)
    at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:198)
    at java.net.http/jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:271)
    at java.net.http/jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:224)
    at java.net.http/jdk.internal.net.http.SocketTube$InternalReadPublisher$InternalReadSubscription.signalReadable(SocketTube.java:763)
    at java.net.http/jdk.internal.net.http.SocketTube$InternalReadPublisher$ReadEvent.signalEvent(SocketTube.java:941)
    at java.net.http/jdk.internal.net.http.SocketTube$SocketFlowEvent.handle(SocketTube.java:245)
    at java.net.http/jdk.internal.net.http.HttpClientImpl$SelectorManager.handleEvent(HttpClientImpl.java:957)
    at java.net.http/jdk.internal.net.http.HttpClientImpl$SelectorManager.lambda$run$3(HttpClientImpl.java:912)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1540)
    at java.net.http/jdk.internal.net.http.HttpClientImpl$SelectorManager.run(HttpClientImpl.java:912)

So for some reason java parses the content length as negative, it parses the header into an Int apperently and that causes an integer overflow, but only with that URL as the other file is 10GB in size but works perfectly.

And there is no problem downloading that file using some tool like wget or a webbrowser like firefox. And it's not a problem with all files on 1fichier, just some.

Can somone explain what is going on?

My code:

import java.io.File
import java.net.CookieHandler
import java.net.URI
import java.net.http.HttpClient
import java.net.http.HttpRequest
import java.net.http.HttpResponse
import java.nio.ByteBuffer
import java.nio.file.Path
import java.time.Duration
import java.util.concurrent.CompletionStage
import java.util.concurrent.Flow

fun main(args: Array<String>) {
    val dir = File("C:\\Users\\kjh\\Downloads")
    val file = File(dir, "test.bin")

    file.delete()

    val client = HttpClient.newBuilder()
            .version(HttpClient.Version.HTTP_1_1)
            .connectTimeout(Duration.ofSeconds(10))
            .build()

    val request = HttpRequest.newBuilder()
            .url("https://map.usbcraft.net/file/usbpc-downloads/java_http_client.fail")
            //.url("https://speed.hetzner.de/10GB.bin")
            .build()

    println("Starting download...")
    client.send(request) {info ->
        println("Version: ${info.version()} Status code ${info.statusCode()} Got headers back ${info.headers()}")
        println("Content-Length parsed to int: ${info.headers().get("content-length")?.toLong()?.toInt()}")
        val responseHandler = HttpResponse.BodyHandlers.ofFile(file.toPath())
        MySubscriber(responseHandler.apply(info))
    }
    println("Done with download...")
}

class MySubscriber(val subscriber: HttpResponse.BodySubscriber<Path>) : HttpResponse.BodySubscriber<Path> {
    private var counter = 0
    private var total = 0L
    override fun onComplete() {
        println("Total bytes: $total")
        subscriber.onComplete()
    }

    override fun onSubscribe(subscription: Flow.Subscription?) {
        println("onSubscribe")
        subscriber.onSubscribe(subscription)
    }

    override fun onNext(item: MutableList<ByteBuffer>) {
        println("Got data!")
        item.forEach { size ->
            println("I got ${size.remaining()} bytes ($counter)")
            total += size.remaining()
        }
        counter++
        subscriber.onNext(item)
    }

    override fun onError(throwable: Throwable) {
        println("I got an error ${throwable.localizedMessage}")
        subscriber.onError(throwable)
    }

    override fun getBody(): CompletionStage<Path> {
        return subscriber.body
    }
}

fun HttpRequest.Builder.url(url: String) = this.uri(URI.create(url))

Answer 1

This is a bug in the HTTP/1.1 stack of the new HttpClient in Java 11, the bug report for it can be found here: https://bugs.openjdk.java.net/browse/JDK-8212926