parse tables from a PDF document

Question

The PDF in this link (http://www.lenovo.com/psref/pdf/psref450.pdf) contains a number of tables like this:

I'd like to programmatically extract the data and the structure from these tables.

Things I've tried: converting the PDF to HTML using

1. Tika: Unfortunately, the tables are converted to space delimited paragraphs - and some of the strings contain spaces so it's notpossible to split them.
2. Python's PDFMiner: returned an assertion error due to missing fonts. I suspect the HTML would have been similar to the output from Tika,though I'll need to resolve the issue with the missing fonts to confirm this.
3. Online tools: I tried http://www.zamzar.com/ and a couple of others. The file was either too big to process (for the online services) or it generated errors.

I was planning to convert the PDF to HTML and then parse it with BeautifulSoup.

The output could be JSON (e.g. one object per table), XML, or pretty much any format that maintains the structure.

Answer 1

You could try PDFBox. The documentation for that is here:

https://pdfbox.apache.org/1.8/cookbook/textextraction.html

Extend org.apache.pdfbox.pdfviewer.PDFPageDrawer and override the strokePath method. From there you can intercept the drawing operations for horizontal and vertical line segments and use that information to determine the column and row positions. You can set up text regions to determine which numbers/letters/characters are drawn in which region. Since you know the layout of the regions are tabular you'll be able to define tables and tell which column and row the extracted text belongs to using simple algorithms.

Answer 2

@alex-woolford: In general, perfect extraction of data (with or without the same formatting that you see in the PDF) is not always possible, thought it is, to some extent less than 100%. I'm saying this based on having worked on a similar project to yours, earlier. I came across similar issues to what you have, and some research on the Net showed that PDF in general is not a perfectly reversible format, i.e. it is not always possible to recover the text and format from a PDF with 100% accuracy. Sometimes characters even get lost, or transposed, and so on, during the extraction process (using some library). This seems to be due to the very nature of the PDF format and specification. It is not a text-based format. It is a derivative of PostScript and has some weird rules about layout of data. And this is according to official PDF documents, or according to the sites of product companies who have been working with PDF for a long time, and whose products are well known.

If less than perfect accuracy is tolerable, there are some products available (thought I don't know of any for Python, as of now). One is xpdf and another is PDFTextStream. I've used the former, not the latter. xpdf is a C library and also has command-line tools. PDFTextStream is a Java tool/library. It was a paid product earlier, but last I checked, it is now free for single-threaded applications, IIRC.

Even though xpdf is for C and PDFTextStream is for Java, you could call them from Python via XML-RPC or some other distributed computing / cross-language communication approach such as sockets. Some work would be involved, for that, of course.

HTH.