URLresponse is not retrieved after storing in cache using storeCachedResponse

Question

Goal

I'm trying to inject data/response from URLRequest into another URLRequest in my cache.

Setup

This is just a sample code. It's ready to be dumped into a project.

What I'm trying to do is use the response + data retrieved from my landscapeURLString network request...store into my session's cache for my lizardURLString request.

import UIKit

class ViewController: UIViewController {

    lazy var defaultSession : URLSession = {
        let urlCache = URLCache(memoryCapacity: 500 * 1024 * 1024, diskCapacity: 500 * 1024 * 1024, diskPath: "something")
        let configuration = URLSessionConfiguration.default
        configuration.urlCache = urlCache
        let session = URLSession(configuration: configuration)

        return session
    }()
    lazy var downloadLizzardbutton : UIButton = {
        let btn = UIButton()
        btn.translatesAutoresizingMaskIntoConstraints = false
        btn.setTitle("download lizard image OFFLINE", for: .normal)
        btn.backgroundColor = .blue
        btn.addTarget(self, action: #selector(downloadLizardAction), for: .touchUpInside)
        return btn
    }()

    let imageView : UIImageView = {
        let imageView = UIImageView()
        imageView.translatesAutoresizingMaskIntoConstraints = false
        imageView.contentMode = .scaleAspectFill
        return imageView
    }()

    // I make sure my internet is set to OFF so that it forces this to be read from cache...
    @objc func downloadLizardAction() {
        downloadImage(from: lizardURLString, from: defaultSession)
    }
    let lizardURLString = "https://upload.wikimedia.org/wikipedia/commons/e/e0/Large_Scaled_Forest_Lizard.jpg"
    let landscapeURLString = "https://images.pexels.com/photos/414171/pexels-photo-414171.jpeg"        

    override func viewDidLoad() {
        super.viewDidLoad()
        view.addSubview(imageView)
        view.addSubview(downloadLizzardbutton)
        imageView.pinToAllEdges(of: view)

        downloadImage(from: landscapeURLString, from: defaultSession)
    }
    private func downloadImage(from urlString: String, from session : URLSession){
        guard let url = URL(string: urlString) else{
            fatalError("bad String we got!")
        }

        let urlRequest = URLRequest(url: url, cachePolicy: .useProtocolCachePolicy, timeoutInterval: 15)
        print("url.hashValue: \(urlRequest.hashValue)")

        let task = session.dataTask(with: urlRequest) { [weak self] (data, response, error) in

            guard error == nil else {
                print(error)
                return
            }
            guard let httpResponse = response as? HTTPURLResponse,
                (200...299).contains(httpResponse.statusCode) else {
                    print("response NOT 2xx: \(response)")
                    return
            }

            for header in httpResponse.allHeaderFields{
                if let key = header.key as? String, key == "Cache-Control"{
                    print("found Cache-Control: \(httpResponse.allHeaderFields["Cache-Control"])")
                }
            }

            if let data = data,
                let image = UIImage(data: data){
                let lizardURL = URL(string: self!.lizardURLString)
                let lizardURLRequest = URLRequest(url: lizardURL!)

                let landscapeCachedURLPResponse : CachedURLResponse = CachedURLResponse(response: response!, data: data, userInfo:nil, storagePolicy: .allowed)
                print("before storing into cache: \(String(describing: session.configuration.urlCache?.cachedResponse(for: lizardURLRequest)))")

                session.configuration.urlCache?.storeCachedResponse(landscapeCachedURLPResponse, for: lizardURLRequest)    

                print("after storing into cache: \(String(describing: session.configuration.urlCache?.cachedResponse(for: lizardURLRequest)))")
                print("lizardRequest.hashValue: \(lizardURLRequest.hashValue)")

                DispatchQueue.main.async {
                    self?.imageView.image = image
                }
            }
        }
        task.resume()
    }        
}


extension UIView{

    func pinToAllEdges(of view: UIView){
        let leading = leadingAnchor.constraint(equalTo: view.leadingAnchor)
        let top = topAnchor.constraint(equalTo: view.topAnchor)
        let trailing = trailingAnchor.constraint(equalTo: view.trailingAnchor)
        let bottom = bottomAnchor.constraint(equalTo: view.bottomAnchor)

        NSLayoutConstraint.activate([leading, top, trailing, bottom])
    }
}

Things I've already validated:

• My landscapeURLString has a cache-control header with a max-age of 31536000
• If it's a fresh install, then before storing into the cache, my cachedResponse for lizardURLString is nil. But after storing, it's no longer nil. As a result I conclude that I'm successfully storing something into the cache!
• I also suspect the URLCache considers the URLRequest as the key. So I printed the hashValue of my lizardURLString. It's same as the key I've stored. Combining that with the point above, I concluded that exact key exists in cache!
• I can also see that when I store it in my cache, my currentMemoryUsage increases.

How I'm testing and what I'm seeing:

1. I just download the landscape image.
2. Turn off my internet
3. Click the button to download the lizard image.

Obviously it's offline. I expect it to use from the cache but it doesn't. All I get is a time out!

I also tried changing the cachePolicy to returnCacheDataElseLoad, but that didn't help either

EDIT1:

I also tried doing what David said and do:

let landscapeHTTPResponse : HTTPURLResponse = HTTPURLResponse(url: self!.lizardURL, statusCode: 200, httpVersion: "HTTP/1.1", headerFields: (httpResponse.allHeaderFields as! [String : String]))!
let landscapedCachedURLPResponse : CachedURLResponse = CachedURLResponse(response: landscapeHTTPResponse, data: data, userInfo:nil, storagePolicy: .allowed)

and the stored landscapedCachedURLPResponse into the cache. That didn't work either. It times out as well — it doesn't every look into the cache.

EDIT2:

So I made some progress. Or perhaps took one step back and one step forward.

I tried to see if I can store the response for the same URL and see if I can retrieve the response after I empty my cache. I wasn't able to.

I was creating my cached response like this:

let cachedResponse = CachedURLResponse(response: response!, data: data, userInfo:nil, storagePolicy: .allowed)

or just like this:

let cachedResponse = CachedURLResponse(response: response!, data: data)

What got this part to work?:

let cachedResponseFromCache = session.configuration.urlCache?.cachedResponse(for: self!.landscapeURLRequest)
self._cachedResponse = cachedResponseFromCache

Then I:

1. flushed the cache
2. turned off internet
3. attempted to download image, but had no success which is good. It's the expected behavior
4. stored cachedResponseFromCache property into the cache.
5. was able to retrieve from cache!

I'm not sure what's the difference between pulling off from cache itself and creating the cache from Response + Data.

This is important because I was starting to question if there are still some form of internal bugs in URLCache. This has given me reason to believe that it may be working as expected.

Now I know the process of storing into cache works. I know my URLResponse is good. I just need to work my way through mapping the URLRequest

EDIT3:

Guy Kogus suggested that my URLs need to be from the same source. So once I downloaded the bearImage he mentioned, my lizardImage was coming through. VOILA!

As very important debugging note that I learned: Even if your getting success on some part (that it was caching the landscape image for itself) of the problem, changing variables (here changing the initial URL) can always change the entire testing results.

He suspected that it was because the Server in header in shared and that's important for looking up the cachedResponse.

I refuted that claim by saying that my lizardURLRequest is made when it's online so there's nothing for it compare with, yet it works! So the next idea was that it may have something to do with some part of the URL, like it's first segment or something.

So then I went and altered the lizardURL from:

https://upload.wikimedia.org/wikipedia/commons/e/e0/Large_Scaled_Forest_Lizard.jpg

to something like: https://skdhfsupload.qwiklkjlkjimedia.com/qwikipehkjdia/eeeeeecommons/sdalfjkdse/aldskfjae0/extraParam/anotherextraparam/asasdLarge_Scaled_Forest_Lizard.jpeg

I added dumb characters in the URL. I also added extra segments into it. I changed the file type at the end.

Still it was working. So the only thing I can conclude is that something from the Headers is doing the decision making.

The headers for my landscapeURL are: (caching for another URL doesn't work for this)

Content-Length : 997361
x-cache : HIT, MISS
cf-ray : 472793e93ce39574-IAD
x-served-by : cache-lax8621-LAX, cache-iad2132-IAD
cf-cache-status : HIT
Last-Modified : Sun, 14 Oct 2018 2:10:05 GMT
Accept-Ranges : bytes
Vary : Accept-Encoding
x-content-type-options : nosniff
Content-Type : image/jpeg
expect-ct : max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Set-Cookie : __cfduid=d5f5fd59ce5ff9ac86e42f8c008708ae61541004176; expires=Thu, 31-Oct-19 16:42:56 GMT; path=/; domain=.pexels.com; HttpOnly
Expires : Thu, 31 Oct 2019 16:42:56 GMT
Server : cloudflare
Cache-Control : public, max-age=31536000
Date : Wed, 31 Oct 2018 16:42:56 GMT

The headers for my BearURL are: (caching for another URL works for this)

Date : Wed, 31 Oct 2018 16:46:38 GMT
Content-Length : 215104
x-client-ip : 2001:558:1400:4e:808c:2738:43e:36f5
access-control-expose-headers : Age, Date, Content-Length, Content-Range, X-Content-Duration, X-Cache, X-Varnish
x-cache : cp1076 miss, cp1088 hit/21
Age : 27646
Etag : 00e21950bf432476c91b811bb685b6af
Strict-Transport-Security : max-age=106384710; includeSubDomains; preload
x-analytics : https=1;nocookies=1
Accept-Ranges : bytes
x-object-meta-sha1base36 : 42tq5grg9rq1ydmqd4z5hmmqj6h2309
x-varnish : 48388488, 503119619 458396839
x-cache-status : hit-front
Content-Type : image/jpeg
x-trans-id : tx08ed43bbcc1946269a9a3-005bd97070
Last-Modified : Fri, 04 Oct 2013 23:30:08 GMT
Access-Control-Allow-Origin : *
timing-allow-origin : *
x-timestamp : 1380929407.39127
Via : 1.1 varnish (Varnish/5.1), 1.1 varnish (Varnish/5.1)

Important note:

For the BearURL, caching for the BearURL and lizardURL or any other URL works. For landscapeURL, caching only works for the landscapeURL itself. It doesn't work for any other URL.

So the current state of the question is: What headers need to be included for this to work?

Answer 1

Welcome to the wonderful world of asynchronous caches. NSURLCache is highly asynchronous. Just because you've shoved data into it doesn't mean it is available for retrieval. You have to let the main run loop return before it will be available, and possibly even wait a little while. The failure to return a response immediately after storing it is not at all unusual. Try dispatching it after five seconds or so.

Second, your cache might be a bit on the small size for storing multi-megabyte images. Try bumping that up and see if it helps.

Finally, what do you mean when you say that you "turn off your Internet?" You say that you're getting a timeout. Normally, if you put the device into Airplane mode with all connectivity disabled, it should not sit there for any significant amount of time before failing with an error indicating no connectivity). If that isn't happening, something strange is happening, almost as if waitsForConnectivity is getting set on the session or something. (You aren't making the network requests in the background, are you? If so, try explicitly setting waitsForConnectivity to NO so that they won't wait for a connection to be available.)

Also, for this usage, you may have to strip out the Vary: Accept-Encoding header or provide a consistent user agent string. That header causes the cache to basically be per-browser. This may cause the cache to behave in unexpected ways, and is probably the cause of the weirdness you're seeing.

Note that stripping out the Vary header is a bit of a hack, and probably isn't the most correct way to fix the issue; ideally, you should tweak whatever outgoing header fields you have to tweak so that it works even with that header present. But you'd have to research it and figure out exactly what fields are needed, because I have no idea. :-)

Answer 2

This isn't a complete answer, but it should put you in the right direction.

The issue isn't to do with your code, I believe that it's mostly fine. The issue is regarding the response that you get from the landscapeURLString because the image is stored in Cloudflare. If you use 2 images from the same source, (e.g. try with this bear from wikipedia instead of the image from images.pexels.com) it should work.

I tried printing out the response and headers of downloading the images.pexels.com image and this is what I saw:

response: <NSHTTPURLResponse: 0x600002bf65c0> { URL: https://upload.wikimedia.org/wikipedia/commons/e/e0/Large_Scaled_Forest_Lizard.jpg } { Status Code: 200, Headers {
    "Accept-Ranges" =     (
        bytes
    );
    "Cache-Control" =     (
        "public, max-age=31536000"
    );
    "Content-Length" =     (
        997361
    );
    "Content-Type" =     (
        "image/jpeg"
    );
    Date =     (
        "Wed, 31 Oct 2018 11:38:52 GMT"
    );
    Expires =     (
        "Thu, 31 Oct 2019 11:38:52 GMT"
    );
    "Last-Modified" =     (
        "Fri, 26 Oct 2018 6:31:56 GMT"
    );
    Server =     (
        cloudflare
    );
    Vary =     (
        "Accept-Encoding"
    );
    "cf-cache-status" =     (
        HIT
    );
    "cf-ray" =     (
        "4725d67b0ae461bd-BCN"
    );
    "expect-ct" =     (
        "max-age=604800, report-uri=\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\""
    );
    "x-cache" =     (
        "HIT, MISS"
    );
    "x-content-type-options" =     (
        nosniff
    );
    "x-served-by" =     (
        "cache-lax8643-LAX, cache-mad9437-MAD"
    );
} }
headers: ["Accept-Ranges": "bytes", "Content-Type": "image/jpeg", "Last-Modified": "Fri, 26 Oct 2018 6:31:56 GMT", "Vary": "Accept-Encoding", "cf-ray": "4725d67b0ae461bd-BCN", "Date": "Wed, 31 Oct 2018 11:38:52 GMT", "Server": "cloudflare", "Expires": "Thu, 31 Oct 2019 11:38:52 GMT", "x-content-type-options": "nosniff", "expect-ct": "max-age=604800, report-uri=\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\"", "x-cache": "HIT, MISS", "x-served-by": "cache-lax8643-LAX, cache-mad9437-MAD", "cf-cache-status": "HIT", "Content-Length": "997361", "Cache-Control": "public, max-age=31536000"]

There's probably something in there that is trying to match the request URL with a response field that's causing the cache miss, but I'm not knowledgable enough to know what it is. Somebody else can probably catch it for you (hence why I said that this answer is incomplete).