Open scrapy output in browser tab or ipython window

Question

I'm working on my first spider after doing the scrapy tut:

class My_Spider(scrapy.Spider):

    name = "My_Spider"


    def start_requests(self):
        for i in range(1):
            yield Request('my_url', method="post", headers= headers, body=payload, callback=self.parse_method)


    def parse_method(self,response):
        print(response.body)

Is there a way to open the response body in a browser window tab or ipython notebook?

Answer 1

Yes, there is an open_in_browser() built into Scrapy utility function:

from scrapy.utils.response import open_in_browser

open_in_browser(response)

Answer 2

You can use Open in browser
Example from the documentation:

from scrapy.utils.response import open_in_browser

def parse_details(self, response):
    if "item name" not in response.body:
        open_in_browser(response)

For your problem it will be like:

from scrapy.utils.response import open_in_browser
open_in_browser(response)

Also, we can see a web page use scrapy shell.
For example:

scrapy shell https://stackoverflow.com/questions/37465387/open-scrapy-output-in-browser-tab-or-ipython-window

and then view the response in a browser

view(response)

Answer 3

To open the response object in a specific browser

import webbrowser
from scrapy.utils.response import open_in_browser

open_in_browser(response, _openfunc=webbrowser.get('/path/to/browser/exe').open)