Can I prevent ABCpdf from mashing words together (e.g. mashingwordstogether) when convertering PDF to Text?

Question

I'm using ABCpdf to extract the text content of some PDF files, in particular by calling Doc.GetText("Text"). (You call it in a loop, once per page.) This usually works well, but for some PDF files the resulting text consists of text with a dearth of space characters, e.g.

Thissentencedoesn'thaveanyspacesbetweenwords.

What's interesting is if I try to extract text from the exact same PDFs using Apache Tika (powered under the hood by PDFBox), I tend to get all the spaces I'd expect between words. That is, the above sentence would be rendered by Tika as

This sentence doesn't have any spaces between words.

Overall, the two tools act like they're afraid of committing different mistakes -- ABCpdf acts like the worst thing in the world would be to insert a space where one doesn't belong, while Tika acts like the worst thing in the world would be to fail to insert a space where one does belong.

Are there any settings to make ABCpdf act more like Tika in this regard?

Answer 1

Short Answer: You can get individual tokens of text via Doc.GetText("SVG"), parsing the XML for TEXT and TSPAN elements, and determining if there is layout spacing that should be treated as actual spaces. The behavior you're seeing from PDFBox is probably their attempt to make that assumption. Also, even Adobe Acrobat can return spaced text via the clipboard as PDFBox does.

Long Answer: This may cause more problems, as this may not be the original intent of the text in the PDF.

ABCpdf is doing the correct thing here, as the PDF spec only describes where things should be placed in the output medium. One can construct a PDF file that ABCpdf interprets in both styles, even though the original sentence looks nearly the same.

To demonstrate this, here is a snapshot of a document from Adobe InDesign that shows a text layout matching both cases for your sample sentence.

Note that the first row was not constructed with actual spaces, instead, the words were placed by hand in individual text regions and lined up to look approximately like a properly spaced sentence. The second row has a single sentence that has actual text spaces between the words, in a single text region.

When exported to PDF and then read in by ABCpdf, Doc.GetText("TEXT") will return the following:

ThisSentenceDoesn'tHaveAnySpacesBetweenWords.  
This Sentence Doesn't Have Any Spaces Between Words.

Thus if you wish to detect layout spaces, you must use SVG output and step through the tokens of text manually. Doc.GetText("SVG") returns text and other drawing entities as ABCpdf sees them on the page, and you can decide how you want to handle the case of layout based spacing.

You'll receive output similar to this:

<?xml version="1.0" standalone="no"?>
<svg width="612" height="792" x="0" y="0" version="1.1" baseProfile="full" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<text xml:space="preserve" x="36" y="46.1924" font-size="14" font-family="ArialMT" textLength="26.446" transform="translate(36, 46.1924) translate(-36, -46.1924)">This</text>
<text xml:space="preserve" x="66.002" y="46.1924" font-size="14" font-family="ArialMT" textLength="59.15" transform="translate(66.002, 46.1924) translate(-66.002, -46.1924)">Sentence</text>
<text xml:space="preserve" x="129.604" y="46.1924" font-size="14" font-family="ArialMT" textLength="47.46" transform="translate(129.604, 46.1924) translate(-129.604, -46.1924)">Doesn&#8217;t</text>
<text xml:space="preserve" x="181.208" y="46.1924" font-size="14" font-family="ArialMT" textLength="32.676" transform="translate(181.208, 46.1924) translate(-181.208, -46.1924)">Have</text>
<text xml:space="preserve" x="219.61" y="46.1924" font-size="14" font-family="ArialMT" textLength="24.122" transform="translate(219.61, 46.1924) translate(-219.61, -46.1924)">Any</text>
<text xml:space="preserve" x="249.612" y="46.1924" font-size="14" font-family="ArialMT" textLength="46.69" transform="translate(249.612, 46.1924) translate(-249.612, -46.1924)">Spaces</text>
<text xml:space="preserve" x="301.216" y="46.1924" font-size="14" font-family="ArialMT" textLength="54.474" transform="translate(301.216, 46.1924) translate(-301.216, -46.1924)">Between</text>
<text xml:space="preserve" x="360.016" y="46.1924" font-size="14" font-family="ArialMT" transform="translate(360.016, 46.1924) translate(-360.016, -46.1924)"><tspan textLength="13.216">W</tspan><tspan dx="-0.252" textLength="31.122">ords.</tspan></text>
<text xml:space="preserve" x="36.014" y="141.9944" font-size="14" font-family="ArialMT" transform="translate(36.014, 141.9944) translate(-36.014, -141.9944)">
<tspan textLength="181.3">This Sentence Doesn&#8217;t Have </tspan><tspan dx="-0.756" textLength="150.178">Any Spaces Between W</tspan><tspan dx="-0.252" textLength="31.122">ords.</tspan></text>
</svg>

And note that the basic structure reveals the original intent that gave you problems. (xml:space and attributes removed, whitespace modifications for the sake of example)

<?xml version="1.0" standalone="no"?>
<svg>
  <text>This</text>
  <text>Sentence</text>
  <text>Doesn&#8217;t</text>
  <text>Have</text>
  <text>Any</text>
  <text>Spaces</text>
  <text>Between</text>
  <text><tspan>W</tspan><tspan>ords.</tspan></text>
  <text>
    <tspan>This Sentence Doesn&#8217;t Have </tspan>
    <tspan>Any Spaces Between W</tspan>
    <tspan>ords.</tspan>
  </text>
</svg>