Fatal Exception: java.lang.OutOfMemoryError using okhttp3 okio for download files

Question

I have a App that downlaod content from the web. music, videos, pdfs.... like a download manager.

But now its is crashing everytime is it downloading content:

E/LVN/advanced_memory_manager.c: ---------------------------------- AMM report ------------------------------
-> Memory Currently Allocated: 0 bytes <=> 0 components
-> Max Memory Need: 512000 bytes
-> Overall Memory Allocation: 515652 bytes (l:423)
E/art: Throwing OutOfMemoryError "Failed to allocate a 2060 byte allocation with 16777232 free bytes and 308MB until OOM; failed due to fragmentation (required continguous free 4096 bytes where largest contiguous free 0 bytes)"
E/AndroidRuntime: FATAL EXCEPTION: OkHttp Dispatcher
    Process: com.rokki.life2, PID: 32171
    java.lang.OutOfMemoryError: Failed to allocate a 2060 byte allocation with 16777232 free bytes and 308MB until OOM; failed due to fragmentation (required continguous free 4096 bytes where largest contiguous free 0 bytes)
        at okio.Segment.<init>(Segment.java:58)
        at okio.SegmentPool.take(SegmentPool.java:46)
        at okio.Buffer.writableSegment(Buffer.java:1114)
        at okio.Okio$2.read(Okio.java:137)
        at okio.AsyncTimeout$2.read(AsyncTimeout.java:211)
        at okio.RealBufferedSource.read(RealBufferedSource.java:50)
        at okhttp3.internal.http.Http1xStream$FixedLengthSource.read(Http1xStream.java:381)
        at okio.RealBufferedSource.request(RealBufferedSource.java:71)
        at okhttp3.logging.HttpLoggingInterceptor.intercept(HttpLoggingInterceptor.java:225)
        at okhttp3.RealCall$ApplicationInterceptorChain.proceed(RealCall.java:187)
        at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:160)
        at okhttp3.RealCall.access$100(RealCall.java:30)
        at okhttp3.RealCall$AsyncCall.execute(RealCall.java:127)
        at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
        at java.lang.Thread.run(Thread.java:818)

and

Fatal Exception: java.lang.OutOfMemoryError: Failed to allocate a 106405148 byte allocation with 16777120 free bytes and 82MB until OOM
       at java.lang.String.<init>(String.java:332)
       at java.lang.String.<init>(String.java:371)
       at okio.Buffer.readString(Buffer.java:579)
       at okio.Buffer.readString(Buffer.java:562)
       at okhttp3.logging.HttpLoggingInterceptor.intercept(HttpLoggingInterceptor.java:244)
       at okhttp3.RealCall$ApplicationInterceptorChain.proceed(RealCall.java:187)
       at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:160)
       at okhttp3.RealCall.access$100(RealCall.java:30)
       at okhttp3.RealCall$AsyncCall.execute(RealCall.java:127)
       at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
       at java.lang.Thread.run(Thread.java:818)

I already added this to my Manifest:

<application
    android:name=".app.MyApp"
    android:allowBackup="true"
    android:hardwareAccelerated="true"
    android:icon="@mipmap/ic_launcher"
    tools:replace="android:icon"
    android:label="@string/app_name"
    android:largeHeap="true"
    android:supportsRtl="true"
    android:theme="@style/AppTheme.NoActionBar">

Code SAMPLE:

private OkHttpClient client;


onViewCreated...{

    //Initiate OkHttp with interceptor
    HttpLoggingInterceptor logging = new HttpLoggingInterceptor();
    logging.setLevel(HttpLoggingInterceptor.Level.BODY);

    client = new OkHttpClient.Builder()
            .addInterceptor(logging)
            .build();

...
}



private void downloadPdf() {
AsyncTask.execute(new Runnable() {
    @Override
    public void run() {


    client
        .newCall(getRequest(Config._API_PDF))
        .enqueue(new Callback() {
            @Override
            public void onFailure(Call call, IOException e) {
                ...
            }


              @Override
                public void onResponse(Call call, Response response) throws IOException {


              try {

                                    InputStream ins = response.body().byteStream();
                                    BufferedReader in = new BufferedReader(new InputStreamReader(ins));

                                    String lineTotal = "";
                                    while (true) {
                                        String line = in.readLine();
                                        if (line == null)
                                            break;
                                        else
                                            lineTotal += line;
                                    }

                                    ...json parsing ...



...

My BuildGradle:

apply plugin: 'com.android.application'

android {
    compileSdkVersion 23
    buildToolsVersion "23.0.2"

    defaultConfig {
        applicationId "com.my.app"
        minSdkVersion 17
        targetSdkVersion 22
        versionCode 1
        versionName "1.0"

        ndk { abiFilters "armeabi", "x86", "mips" }
    }
    buildTypes {
        release {
            minifyEnabled false
            proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
        }
    }
}


dependencies {
    compile fileTree(dir: 'libs', include: ['*.jar'])

    final SUPPORT_LIBRARY_VERSION = '23.2.0'
    final PLAY_SERVICES_VERSION = '8.3.0'
    final RETROFIT_VERSION = '2.0.0'
    final OKHTTP3_VERSION = '3.2.0'


    compile "com.android.support:design:$SUPPORT_LIBRARY_VERSION"
    compile "com.android.support:support-v4:$SUPPORT_LIBRARY_VERSION"
    compile "com.android.support:appcompat-v7:$SUPPORT_LIBRARY_VERSION"
    compile "com.android.support:cardview-v7:$SUPPORT_LIBRARY_VERSION"
    compile "com.android.support:recyclerview-v7:$SUPPORT_LIBRARY_VERSION"


    compile "com.squareup.retrofit2:retrofit:$RETROFIT_VERSION"
    compile "com.squareup.retrofit2:converter-gson:$RETROFIT_VERSION"
    compile "com.squareup.retrofit2:adapter-rxjava:$RETROFIT_VERSION"
    compile "com.squareup.okhttp3:logging-interceptor:$OKHTTP3_VERSION"
    compile 'com.google.code.gson:gson:2.6.2'
    compile 'com.jakewharton:butterknife:7.0.1'

}

then I have another similar method for download Video...

Does anyone came across this before ? Thanks

Answer 1

When you use HttpLoggingInterceptor.Level.BODY , then you try to download large file , will save all body in memory for log.

That's easy let to OOM .

Try to remove log body or just logging NONE , basic or header and try again.

And when you try to get large file can use @Streaming in retrofit2 like this.

@Streaming
@GET
fun downloadFileWithDynamicUrlSync(@Url fileUrl: String): Call<ResponseBody>

Try to remove log body in here.

onViewCreated...{

//Initiate OkHttp with interceptor
HttpLoggingInterceptor logging = new HttpLoggingInterceptor();
logging.setLevel(HttpLoggingInterceptor.Level.BODY);

Try this .

onViewCreated...{

//Initiate OkHttp with interceptor
HttpLoggingInterceptor logging = new HttpLoggingInterceptor();
logging.setLevel(HttpLoggingInterceptor.Level.NONE);

Answer 2

If in hurry copy/paste this version of HttpLoggingInterceptor and use it just like original. It will solve your issue by cutting off logs that are too long. Works really nice.

class CustomHttpLoggingInterceptor @JvmOverloads constructor(
        private val maxLogSize : Long = 5000,
        private val logger: Logger = Logger.DEFAULT
) : Interceptor {

    @Volatile private var headersToRedact = emptySet<String>()

    @set:JvmName("level")
    @Volatile var level = Level.NONE

    enum class Level {
        /** No logs. */
        NONE,

        /**
         * Logs request and response lines.
         *
         * Example:
         * ```
         * --> POST /greeting http/1.1 (3-byte body)
         *
         * <-- 200 OK (22ms, 6-byte body)
         * ```
         */
        BASIC,

        /**
         * Logs request and response lines and their respective headers.
         *
         * Example:
         * ```
         * --> POST /greeting http/1.1
         * Host: example.com
         * Content-Type: plain/text
         * Content-Length: 3
         * --> END POST
         *
         * <-- 200 OK (22ms)
         * Content-Type: plain/text
         * Content-Length: 6
         * <-- END HTTP
         * ```
         */
        HEADERS,

        /**
         * Logs request and response lines and their respective headers and bodies (if present).
         *
         * Example:
         * ```
         * --> POST /greeting http/1.1
         * Host: example.com
         * Content-Type: plain/text
         * Content-Length: 3
         *
         * Hi?
         * --> END POST
         *
         * <-- 200 OK (22ms)
         * Content-Type: plain/text
         * Content-Length: 6
         *
         * Hello!
         * <-- END HTTP
         * ```
         */
        BODY
    }

    interface Logger {
        fun log(message: String)

        companion object {
            /** A [Logger] defaults output appropriate for the current platform. */
            @JvmField
            val DEFAULT: Logger = object : Logger {
                override fun log(message: String) {
                    Platform.get().log(Platform.INFO, message, null)
                }
            }
        }
    }

    fun redactHeader(name: String) {
        val newHeadersToRedact = TreeSet(String.CASE_INSENSITIVE_ORDER)
        newHeadersToRedact += headersToRedact
        newHeadersToRedact += name
        headersToRedact = newHeadersToRedact
    }

    @Deprecated(
            message = "Moved to var. Replace setLevel(...) with level(...) to fix Java",
            replaceWith = ReplaceWith(expression = "apply { this.level = level }"),
            level = DeprecationLevel.WARNING)
    fun setLevel(level: Level) = apply {
        this.level = level
    }

    @JvmName("-deprecated_level")
    @Deprecated(
            message = "moved to var",
            replaceWith = ReplaceWith(expression = "level"),
            level = DeprecationLevel.ERROR)
    fun getLevel(): Level = level

    @Throws(IOException::class)
    override fun intercept(chain: Interceptor.Chain): Response {
        val level = this.level

        val request = chain.request()
        if (level == Level.NONE) {
            return chain.proceed(request)
        }

        val logBody = level == Level.BODY
        val logHeaders = logBody || level == Level.HEADERS

        val requestBody = request.body

        val connection = chain.connection()
        var requestStartMessage =
                ("--> ${request.method} ${request.url}${if (connection != null) " " + connection.protocol() else ""}")
        if (!logHeaders && requestBody != null) {
            requestStartMessage += " (${requestBody.contentLength()}-byte body)"
        }
        logger.log(requestStartMessage)

        if (logHeaders) {
            if (requestBody != null) {
                // Request body headers are only present when installed as a network interceptor. Force
                // them to be included (when available) so there values are known.
                requestBody.contentType()?.let {
                    logger.log("Content-Type: $it")
                }
                if (requestBody.contentLength() != -1L) {
                    logger.log("Content-Length: ${requestBody.contentLength()}")
                }
            }

            val headers = request.headers
            for (i in 0 until headers.size) {
                val name = headers.name(i)
                // Skip headers from the request body as they are explicitly logged above.
                if (!"Content-Type".equals(name, ignoreCase = true) &&
                        !"Content-Length".equals(name, ignoreCase = true)) {
                    logHeader(headers, i)
                }
            }

            if (!logBody || requestBody == null) {
                logger.log("--> END ${request.method}")
            } else if (bodyHasUnknownEncoding(request.headers)) {
                logger.log("--> END ${request.method} (encoded body omitted)")
            } else if (requestBody.isDuplex()) {
                logger.log("--> END ${request.method} (duplex request body omitted)")
            } else {
                val buffer = Buffer()
                requestBody.writeTo(buffer)

                val contentType = requestBody.contentType()
                val charset: Charset = contentType?.charset(StandardCharsets.UTF_8) ?: StandardCharsets.UTF_8

                logger.log("")
                if (buffer.isProbablyUtf8()) {
                    if(requestBody.contentLength()>maxLogSize)logger.log(buffer.readString(maxLogSize,charset))
                    else logger.log(buffer.readString(charset))
                    logger.log("--> END ${request.method} (${requestBody.contentLength()}-byte body)")
                } else {
                    logger.log(
                            "--> END ${request.method} (binary ${requestBody.contentLength()}-byte body omitted)")
                }
            }
        }

        val startNs = System.nanoTime()
        val response: Response
        try {
            response = chain.proceed(request)
        } catch (e: Exception) {
            logger.log("<-- HTTP FAILED: $e")
            throw e
        }

        val tookMs = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startNs)

        val responseBody = response.body!!
        val contentLength = responseBody.contentLength()
        val bodySize = if (contentLength != -1L) "$contentLength-byte" else "unknown-length"
        logger.log(
                "<-- ${response.code}${if (response.message.isEmpty()) "" else ' ' + response.message} ${response.request.url} (${tookMs}ms${if (!logHeaders) ", $bodySize body" else ""})")

        if (logHeaders) {
            val headers = response.headers
            for (i in 0 until headers.size) {
                logHeader(headers, i)
            }

            if (!logBody || !response.promisesBody()) {
                logger.log("<-- END HTTP")
            } else if (bodyHasUnknownEncoding(response.headers)) {
                logger.log("<-- END HTTP (encoded body omitted)")
            } else {
                val source = responseBody.source()
                source.request(Long.MAX_VALUE) // Buffer the entire body.
                var buffer = source.buffer

                var gzippedLength: Long? = null
                if ("gzip".equals(headers["Content-Encoding"], ignoreCase = true)) {
                    gzippedLength = buffer.size
                    GzipSource(buffer.clone()).use { gzippedResponseBody ->
                        buffer = Buffer()
                        buffer.writeAll(gzippedResponseBody)
                    }
                }

                val contentType = responseBody.contentType()
                val charset: Charset = contentType?.charset(StandardCharsets.UTF_8) ?: StandardCharsets.UTF_8

                if (!buffer.isProbablyUtf8()) {
                    logger.log("")
                    logger.log("<-- END HTTP (binary ${buffer.size}-byte body omitted)")
                    return response
                }

                if (contentLength != 0L) {
                    logger.log("")
                    logger.log(buffer.clone().readString(charset))
                }

                if (gzippedLength != null) {
                    logger.log("<-- END HTTP (${buffer.size}-byte, $gzippedLength-gzipped-byte body)")
                } else {
                    logger.log("<-- END HTTP (${buffer.size}-byte body)")
                }
            }
        }

        return response
    }

    private fun logHeader(headers: Headers, i: Int) {
        val value = if (headers.name(i) in headersToRedact) "██" else headers.value(i)
        logger.log(headers.name(i) + ": " + value)
    }

    private fun bodyHasUnknownEncoding(headers: Headers): Boolean {
        val contentEncoding = headers["Content-Encoding"] ?: return false
        return !contentEncoding.equals("identity", ignoreCase = true) &&
                !contentEncoding.equals("gzip", ignoreCase = true)
    }

    fun  Buffer.isProbablyUtf8(): Boolean {
        try {
            val prefix = Buffer()
            val byteCount = size.coerceAtMost(64)
            copyTo(prefix, 0, byteCount)
            for (i in 0 until 16) {
                if (prefix.exhausted()) {
                    break
                }
                val codePoint = prefix.readUtf8CodePoint()
                if (Character.isISOControl(codePoint) && !Character.isWhitespace(codePoint)) {
                    return false
                }
            }
            return true
        } catch (_: EOFException) {
            return false // Truncated UTF-8 sequence.
        }
    }

}

Then just call it like you would normally:

 
   val interceptor = CustomHttpLoggingInterceptor( )
            interceptor.apply { interceptor.level = CustomHttpLoggingInterceptor.Level.BODY }
            clientBuilder.addInterceptor(interceptor)
 

DEEPER DIVE:

This issue is well know by OkHttp team and they see it as non-issue and recommend for dev to fork project and implement their own versions suited for their needs. Which make sense with their project strategie. read more here

Error accures at this line buffer.readString(charset) that try's to log your entire request body, by limiting it buffer.readString(maxLogSize,charset) we cut off any log lines after that avoding OOM error.Simple!