Downloading csv file from a website using python [duplicate]

Question

I have the following website yahoo finance. I want to set a date range on that page for e.g April 3rd 1997 to 4th November 2015. Once I set the date range I get a link to download the file as csv on that page, below the tables. I want to download the csv file. But I want all of this to be done programmatically. How do I achieve this using python.

Answer 1

This might be helpful :

import requests
import shutil
def callme():
    url = "http://real-chart.finance.yahoo.com/table.csv?s=%5EBSESN&a=03&b=3&c=1997&d=10&e=4&f=2015&g=d&ignore=.csv"
    r = requests.get(url, verify=False,stream=True)
    if r.status_code!=200:
        print "Failure!!"
        exit()
    else:
        r.raw.decode_content = True
        with open("file1.csv", 'wb') as f:
            shutil.copyfileobj(r.raw, f)
        print "Success"

if __name__ == '__main__':
    callme()

How to get this URL ?
You can get the list of API calls in any website by right click-> Inspect Element ->Network.
Now when you make any request from browser, it will list out all the API calls.

You can split the date according to your need and pass it in the URL. You need to do some research about how Yahoo passes the date in URL.

Edit 1: This script will run over HTTP and HTTPS.