Scrapy FormRequest sending JSON

Question

I am trying to create a FormRequest that can send content-type:application/json.

Here is what I try:

yield FormRequest("abc.someurl.com", formdata=json.dumps({"referenceId":123,"referenceType":456}), headers={'content-type':'application/json'}, callback=self.parseResult2)

If I use json.dumps() to process the form data in the formdata=, the error I get is

"exceptions.ValueError: need more than 1 value to unpack"

I can't just use the list as in

formdata={"referenceId":123,"referenceType":456} 

The FormRequest works but is not accepted by the server.

import requests
import json
result = requests.post(url, json.dumps({"referenceId":123,"referenceType":456}), headers={'content-type':'application/json'})

It works from the python command prompt as in the above.

Any ideas?

-KM

Answer 1

from scrapy.http import JsonRequest

class TestSpider(scrapy.Spider):
    def start_requests(self):
        data = {
            'name1': 'value1',
            'name2': 'value2',
        }
        yield JsonRequest(url='http://www.example.com/post/action', data=data)

see more: docs.scrapy.org#jsonrequest

Answer 2

FormRequest is for simulating an HTML form (e.g. application/x-www-form-urlencoded). It sounds like you are simply wanting to POST data with your Request. Since you mention a content type of 'application/json' you probably want to do something like this:

request = Request( url, method='POST', 
                   body=json.dumps(my_data), 
                   headers={'Content-Type':'application/json'} )