Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Convert text data from requests object to dataframe with pandas

Tags:

Using requests I am creating an object which is in .csv format. How can I then write that object to a DataFrame with pandas?

To get the requests object in text format:

import requests import pandas as pd url = r'http://test.url'  r = requests.get(url) r.text  #this will return the data as text in csv format 

I tried (doesn't work):

pd.read_csv(r.text) pd.DataFrame.from_csv(r.text) 
like image 428
sparrow Avatar asked Aug 29 '16 19:08

sparrow


People also ask

Is PyArrow faster than pandas?

There's a better way. It's called PyArrow — an amazing Python binding for the Apache Arrow project. It introduces faster data read/write times and doesn't otherwise interfere with your data analysis pipeline. It's the best of both worlds, as you can still use Pandas for further calculations.

How do you convert Dtype to pandas?

In order to convert data types in pandas, there are three basic options: Use astype() to force an appropriate dtype. Create a custom function to convert the data. Use pandas functions such as to_numeric() or to_datetime()


2 Answers

Try this

import requests import pandas as pd import io  urlData = requests.get(url).content rawData = pd.read_csv(io.StringIO(urlData.decode('utf-8'))) 
like image 117
Merlin Avatar answered Sep 26 '22 02:09

Merlin


I think you can use read_csv with url:

pd.read_csv(url) 

filepath_or_buffer : str, pathlib.Path, py._path.local.LocalPath or any object with a read() method (such as a file handle or StringIO)

The string could be a URL. Valid URL schemes include http, ftp, s3, and file. For file URLs, a host is expected. For instance, a local file could be file ://localhost/path/to/table.csv

import pandas as pd import io import requests  url = r'http://...'  r = requests.get(url)   df = pd.read_csv(io.StringIO(r)) 

If it doesnt work, try update last line:

import pandas as pd import io import requests  url = r'http://...'  r = requests.get(url)   df = pd.read_csv(io.StringIO(r.text)) 
like image 20
jezrael Avatar answered Sep 24 '22 02:09

jezrael