Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Parse text response from http request in Python

I am trying to fetch data from an APU but as response I am getting the plain text. I want to read all text line by line.

This is the url variable: http://www.amfiindia.com/spages/NAVAll.txt?t=23052017073640

First snippet:

from pymongo import MongoClient
import requests
from bs4 import BeautifulSoup as bs
url = "https://www.amfiindia.com/spages/NAVAll.txt?t=23052017073640"
request = requests.get(url)
soup = bs(request.text,"lxml")
for line in soup:
    print line
    break

Result: It prints out the entire text

Second snippet:

request = requests.get(url)
for line in request.text():
    print line
    break

Result: It prints out 1 character

request = requests.get(url)
requestText = request.text()
allMf = requestText.splitlines()

Result: Exception: 'unicode' object is not callable

I have tried few more cases but not able to read text line by line.

like image 527
Jitesh Avatar asked Aug 30 '17 07:08

Jitesh


People also ask

How do I extract data from a response text in Python?

Now, in order to retrieve the data from the response object, we need to convert the raw response content into a JSON type data structure. This is achieved by using json() method. Finally, we extract the required information by parsing down the JSON type object.

What does Response text () do?

text() The text() method of the Response interface takes a Response stream and reads it to completion. It returns a promise that resolves with a String .

What is HTTP response in Python?

The HyperText Transfer Protocol(HTTP) is an application-layer protocol used to transmit hypermedia documents (such as HTML). The request module in Python is used to make HTTP requests to web pages. A web server usually sends an HTTP response back when it receives a request.

What is the type of response content in Python?

content returns the content of the response, in bytes. Basically, it refers to Binary Response content.


2 Answers

request.text is a property and not a method, request.text returns a unicode string, request.text() throws the error 'unicode' object is not callable.

for line in request.text.splitlines():
    print line
like image 146
Danil Speransky Avatar answered Sep 19 '22 17:09

Danil Speransky


import requests
from bs4 import BeautifulSoup as bs
url = "https://www.amfiindia.com/spages/NAVAll.txt?t=23052017073640"
request = requests.get(url)
soup = bs(request.text,"lxml")

# soup.text is to get the returned text
# split function, splits the entire text into different lines (using '\n') and stores in a list. You can define your own splitter.
# each line is stored as an element in the allLines list.
allLines = soup.text.split('\n') 

for line in allLines: # you iterate through the list, and print the single lines
    print(line)
    break # to just print the first line, to show this works
like image 35
SajidSalim Avatar answered Sep 19 '22 17:09

SajidSalim