Using requests I am creating an object which is in .csv format. How can I then write that object to a DataFrame with pandas?
To get the requests object in text format:
import requests import pandas as pd url = r'http://test.url' r = requests.get(url) r.text #this will return the data as text in csv format
I tried (doesn't work):
pd.read_csv(r.text) pd.DataFrame.from_csv(r.text)
There's a better way. It's called PyArrow — an amazing Python binding for the Apache Arrow project. It introduces faster data read/write times and doesn't otherwise interfere with your data analysis pipeline. It's the best of both worlds, as you can still use Pandas for further calculations.
In order to convert data types in pandas, there are three basic options: Use astype() to force an appropriate dtype. Create a custom function to convert the data. Use pandas functions such as to_numeric() or to_datetime()
Try this
import requests import pandas as pd import io urlData = requests.get(url).content rawData = pd.read_csv(io.StringIO(urlData.decode('utf-8')))
I think you can use read_csv
with url
:
pd.read_csv(url)
filepath_or_buffer : str, pathlib.Path, py._path.local.LocalPath or any object with a read() method (such as a file handle or StringIO)
The string could be a URL. Valid URL schemes include http, ftp, s3, and file. For file URLs, a host is expected. For instance, a local file could be file ://localhost/path/to/table.csv
import pandas as pd import io import requests url = r'http://...' r = requests.get(url) df = pd.read_csv(io.StringIO(r))
If it doesnt work, try update last line:
import pandas as pd import io import requests url = r'http://...' r = requests.get(url) df = pd.read_csv(io.StringIO(r.text))
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With