How to click submit button in a form?

Question

I am trying to scrap this website for fund price history, by providing the start date, end date and click the 'Get Prices' button via POST method.

However the page requests.post() return does not contain the results, as if the "Get Price" button was never pressed. This is the URL I put together through the code:

https://www.nysaves.org/nytpl/fundperform/fundHistory.do?submit=Get+Prices&endDate=02%2F20%2F2016&fundid=1003022&startDate=01%2F01%2F2016

I read other posts on Stackoverflow about submitting form via POST in Python and I couldn't make it work. Could you please help? Many thanks!

import requests
import datetime

startDate = datetime.datetime(2016,1,1).strftime('%m/%d/%Y')
endDate = datetime.datetime(2016,2,20).strftime('%m/%d/%Y')

serviceurl = 'https://www.nysaves.org/nytpl/fundperform/fundHistory.do?'
payload = {'fundid':1003022, 'startDate':startDate, 'endDate': endDate, 'submit':'Get Prices'}
r = requests.post(serviceurl, params=payload)

#from IPython.core.display import HTML
#HTML(r.content.decode('utf-8'))

Answer 1

1. You need to use "data", not "params". "params" is used for GET parameters encoded into the URL, but "data" is sent in the body.
2. The correct url is https://www.nysaves.org/nytpl/fundperform/fundHistorySearch.do, not https://www.nysaves.org/nytpl/fundperform/fundHistory.do?.
3. You don't need the submit keyword. But adding it won't hurt.

This code will work:

startDate = datetime.datetime(2016,1,1).strftime('%m/%d/%Y')
endDate = datetime.datetime(2016,2,20).strftime('%m/%d/%Y')

serviceurl = 'https://www.nysaves.org/nytpl/fundperform/fundHistorySearch.do'
payload = {'fundid': 1003022, 'startDate': startDate, 'endDate': endDate}
r = requests.post(serviceurl, data=payload)
print(r.text)