Preserving line breaks when parsing with Scrapy in Python

Question

I've written a Scrapy spider that extracts text from a page. The spider parses and outputs correctly on many of the pages, but is thrown off by a few. I'm trying to maintain line breaks and formatting in the document. Pages such as http://www.state.gov/r/pa/prs/dpb/2011/04/160298.htm are formatted properly like such:

April 7, 2011

Mark C. Toner

2:03 p.m. EDT

MR. TONER: Good afternoon, everyone. A couple of things at the top, and then I’ll take your questions. We condemn the attack on innocent civilians in southern Israel in the strongest possible terms, as well as ongoing rocket fire from Gaza. As we have reiterated many times, there’s no justification for the targeting of innocent civilians, and those responsible for these terrorist acts should be held accountable. We are particularly concerned about reports that indicate the use of an advanced anti-tank weapon in an attack against civilians and reiterate that all countries have obligations under relevant United Nations Security Council resolutions to prevent illicit trafficking in arms and ammunition. Also just a brief statement --

QUESTION: Can we stay on that just for one second?

MR. TONER: Yeah. Go ahead, Matt.

QUESTION: Apparently, the target of that was a school bus. Does that add to your outrage?

MR. TONER: Well, any attack on innocent civilians is abhorrent, but certainly the nature of the attack is particularly so.

While pages like http://www.state.gov/r/pa/prs/dpb/2009/04/121223.htm have output like this with no line breaks:

April 2, 2009

Robert Wood

11:53 a.m. EDTMR. WOOD: Good morning, everyone. I think it’s just about still morning. Welcome to the briefing. I don’t have anything, so – sir.QUESTION: The North Koreans have moved fueling tankers, or whatever, close to the site. They may or may not be fueling this missile. What words of wisdom do you have for the North Koreans at this moment?MR. WOOD: Well, Matt, I’m not going to comment on, you know, intelligence matters. But let me just say again, we call on the North to desist from launching any type of missile. It would be counterproductive. It’s provocative. It further inflames tensions in the region. We want to see the North get back to the Six-Party framework and focus on denuclearization.Yes.QUESTION: Japan has also said they’re going to call for an emergency meeting in the Security Council, you know, should this launch go ahead. Is this something that you would also be looking for?MR. WOOD: Well, let’s see if this test happens. We certainly hope it doesn’t. Again, calling on the North not to do it. But certainly, we will – if that test does go forward, we will be having discussions with our allies.

The code I'm using is as follows:

def parse_item(self, response):
    self.log('Hi, this is an item page! %s' % response.url) 

    hxs = HtmlXPathSelector(response)

    speaker = hxs.select("//span[contains(@class, 'official_s_name')]") #gets the speaker
    speaker = speaker.select('string()').extract()[0] #extracts speaker text
    date = hxs.select('//*[@id="date_long"]') #gets the date
    date = date.select('string()').extract()[0] #extracts the date
    content = hxs.select('//*[@id="centerblock"]') #gets the content
    content = content.select('string()').extract()[0] #extracts the content

    texts = "%s\n\n%s\n\n%s" % (date, speaker, content) #puts everything together in a string

    filename = ("/path/StateDailyBriefing-" + '%s' ".txt") % (date) #creates a file using the date

    #opens the file defined above and writes 'texts' using utf-8
    with codecs.open(filename, 'w', encoding='utf-8') as output:
        output.write(texts)

I think they problem lies in the formatting of the HTML of the page. On the pages that output the text incorrectly, the paragraphs are separated by <br> <p></p>, while on the pages that output correctly the paragraphs are contained within <p align="left" dir="ltr">. So, while I've identified this, I'm not sure how to make everything output consistently in the correct form.

Answer 1

The problem is that when you getting text() or string(), <br> tags are not converted to newline.

Workaround - replace <br> tags before doing XPath requests. Code:

response = response.replace(body=response.body.replace('<br />', '\n')) 
hxs = HtmlXPathSelector(response)

And let me give some advice, if you know, that there is only one node, you can use text() instead string():

date = hxs.select('//*[@id="date_long"]/text()').extract()[0]

Answer 2

Try this xpath:

//*[@id="centerblock"]//text()