Python/Django using python-instagram (400) APISubscriptionError-Unable to reach callback URL

Question

Things used to work. Then it started working occasionally, until it totally stopped working.

Following is my subscription code:

def instagram_realtime_subscribe(event_slug, topic):
    api = InstagramAPI(client_id = CLIENT_ID, client_secret = CLIENT_SECRET)

    r = api.create_subscription(object = 'tag',
                            object_id = topic,
                            aspect = 'media',
                            callback_url = 'http://<domain>/event/%s/import/instagram/realtime'%(event_slug),
                            client_id = CLIENT_ID,
                            client_secret = CLIENT_SECRET
)

Following is my view for handling the GET and POST requests from instagram:

def import_instagram_rt(request, slug):
    if request.method == "GET":
        mode = request.GET.get("hub.mode")
        challenge = request.GET.get("hub.challenge")
        verify_token = request.GET.get("hub.verify_token")
        if challenge:
            return HttpResponse(challenge, mimetype='text/html')
        else:
            return HttpResponse("test", mimetype='text/html')
    else:
        x_hub_signature=''
        if request.META.has_key('HTTP_X_HUB_SIGNATURE'):
            x_hub_signature = request.META['HTTP_X_HUB_SIGNATURE']
        raw_response = request.raw_post_data
        data = simplejson.loads(raw_response)
        for update in data:
            fetch_data(slug, update["object_id"])

Following is my urls.py

url(r'^event/([-\w]+)/import/instagram/realtime$', import_instagram_rt),

This is used to work beautifully. However, it stopped working since two days. Whenever the subscription function is called, it throws the error:

>>> instagram_realtime_subscribe("cats", "cats")
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/ubuntu/webapps/django-projects/imports/views.py", line 687, in instagram_realtime_subscribe
    client_secret = CLIENT_SECRET
  File "/home/ubuntu/webapps/django-projects/local/lib/python2.7/site-packages/instagram/bind.py", line 151, in _call
    return method.execute()
  File "/home/ubuntu/webapps/django-projects/local/lib/python2.7/site-packages/instagram/bind.py", line 143, in execute
    content, next = self._do_api_request(url, method, body, headers)
  File "/home/ubuntu/webapps/django-projects/local/lib/python2.7/site-packages/instagram/bind.py", line 124, in _do_api_request
    raise InstagramAPIError(status_code, content_obj['meta']['error_type'], content_obj['meta']['error_message'])
InstagramAPIError: (400) APISubscriptionError-Unable to reach callback URL "http://<domain>/event/cats/import/instagram/realtime".

I tried hitting the Callback URL manually and got the response "test" which is what you would expect from the function I have written. I tried a requests.get() manually to that url before calling the subscription function and that returned a 200 response.

Why can't Instagram find my callback url when everyone else can access it?

Answer 1

I think import_instagram_rt is not called because they're "Unable to reach callback URL".

Let's break down what happens when they try to reach your callback URL.

They parse the URL

Is your URL well-formed? http://<domain>/event/cats/import/instagram/realtime isn't but I think you're hiding the real hostname. ;-)

They resolve the hostname

Is your DNS ok? Have you done any updates to your zone and messed up the serial? Are the zone transfers between your primary and secondary servers functioning? "working occasionally", may mean that some round-robin scheme is dysfunctional or that some DNS servers give bad replies or timeout.

They open a TCP socket to your IP on port 80

You know the drill: SYN, SYN-ACK, ACK. If you see something slightly off, due to some amusing hardware anomaly, consider sending it to the museum of broken packets.

Seriously... Are all hops in your network routing these packets or are there any firewalls that may be dropping them? Do you have a "helpful" ISP that has blacklists and null-routes?

A process on your machine handles the connection

Do you have a process to accept the connection? If you serve using gunicorn or similar WSGI servers... do you run it behind a reverse proxy, that buffers connections? Otherwise your worker(s) may be busy with instagram_realtime_subscribe for your visitor, and blocking the callback call from Instagram.

That process responds with output

Only until this point your Django app comes into play. Routing the requestto import_instagram_rt, if urls.py is correct...

Happy hunting through logs and tcpdumps...

PS

Why did you remove this from your question?

My heroku log said the following:

2013-11-13T11:30:28.766021+00:00 heroku[router]: at=error code=H12 desc="Request timeout" method=GET path=/instag/realtime_callback/?hub.challenge=b81d6b83b6f14fbf86015670c58e6877&hub.mode=subscribe host=ontodjangoyo.herokuapp.com fwd="54.208.187.11" dyno=web.1 connect=1ms service=30000ms status=503 bytes=0

The status 503 is a strong indication that you don't have a worker available to handle the instagram request... If you have 1 worker and the heroku proxy doesn't buffer requests, then when you "tried hitting the Callback URL manually" the one worker responds as expected. But when your django app is handling a request and calls instagram_realtime_subscribe. then your worker is blocking until it gets a response from instagram, which it doesn't get, because instagram is waiting your worker to become available to handle the callback. This will wait on each other till a timeout occurs. That 503 status may be cached by the proxy for a while.