How to reduce the size of merged PDF/A-1b files with pdfbox or other java library

Question

Input: A list of (e.g. 14) PDF/A-1b files with embedded fonts.
Processing: Doing a simple merge with Apache PDFBOX.
Result: 1 PDF/A-1b file with large (too large) file size. (It is almost the sum of the size of all the source files).

Question: Is there a way to reduce the file size of the resulting PDF?
Idea: Remove redundant embedded fonts. But how to? And is it the right way to do?

Unfortunately the following code is not doing the job, but is highlighting the obvious problem.

try (PDDocument document = PDDocument.load(new File("E:/tmp/16189_ZU_20181121195111_5544_2008-12-31_Standardauswertung.pdf"))) {
    List<COSName> collectedFonts = new ArrayList<>();
    PDPageTree pages = document.getDocumentCatalog().getPages();
    int pageNr = 0;
    for (PDPage page : pages) {
        pageNr++;
        Iterable<COSName> names = page.getResources().getFontNames();
        System.out.println("Page " + pageNr);
        for (COSName name : names) {
            collectedFonts.add(name);
            System.out.print("\t" + name + " - ");
            PDFont font = page.getResources().getFont(name);
            System.out.println(font + ", embedded: " + font.isEmbedded());
            page.getCOSObject().removeItem(COSName.F);
            page.getResources().getCOSObject().removeItem(name);
        }
    }
    document.save("E:/tmp/output.pdf");
}

The code produces an output like that:

Page 1
    COSName{F23} - PDTrueTypeFont ArialMT-Bold, embedded: true
    COSName{F27} - PDTrueTypeFont ArialMT-Regular, embedded: true
Page 2
    COSName{F23} - PDTrueTypeFont ArialMT-Bold, embedded: true
    COSName{F33} - PDTrueTypeFont ArialMT-BoldItalic, embedded: true
    COSName{F25} - PDTrueTypeFont ArialMT-Italic, embedded: true
    COSName{F27} - PDTrueTypeFont ArialMT-Regular, embedded: true
Page 3
    COSName{F23} - PDTrueTypeFont ArialMT-Bold, embedded: true
    COSName{F25} - PDTrueTypeFont ArialMT-Italic, embedded: true
    COSName{F27} - PDTrueTypeFont ArialMT-Regular, embedded: true
Page 4
    COSName{F23} - PDTrueTypeFont ArialMT-Bold, embedded: true
    COSName{F25} - PDTrueTypeFont ArialMT-Italic, embedded: true
    COSName{F27} - PDTrueTypeFont ArialMT-Regular, embedded: true
Page 5
    COSName{F23} - PDTrueTypeFont ArialMT-Bold, embedded: true
    COSName{F33} - PDTrueTypeFont ArialMT-BoldItalic, embedded: true
    COSName{F27} - PDTrueTypeFont ArialMT-Regular, embedded: true
Page 6
    COSName{F23} - PDTrueTypeFont ArialMT-Bold, embedded: true
    COSName{F33} - PDTrueTypeFont ArialMT-BoldItalic, embedded: true
    COSName{F27} - PDTrueTypeFont ArialMT-Regular, embedded: true
Page 7
    COSName{F23} - PDTrueTypeFont ArialMT-Bold, embedded: true
    COSName{F33} - PDTrueTypeFont ArialMT-BoldItalic, embedded: true
    COSName{F27} - PDTrueTypeFont ArialMT-Regular, embedded: true
Page 8
    COSName{F23} - PDTrueTypeFont ArialMT-Bold, embedded: true
    COSName{F25} - PDTrueTypeFont ArialMT-Italic, embedded: true
    COSName{F27} - PDTrueTypeFont ArialMT-Regular, embedded: true
Page 9
    COSName{F23} - PDTrueTypeFont ArialMT-Bold, embedded: true
    COSName{F33} - PDTrueTypeFont ArialMT-BoldItalic, embedded: true
    COSName{F25} - PDTrueTypeFont ArialMT-Italic, embedded: true
    COSName{F27} - PDTrueTypeFont ArialMT-Regular, embedded: true
Page 10
    COSName{F23} - PDTrueTypeFont ArialMT-Bold, embedded: true
    COSName{F33} - PDTrueTypeFont ArialMT-BoldItalic, embedded: true
    COSName{F25} - PDTrueTypeFont ArialMT-Italic, embedded: true
    COSName{F27} - PDTrueTypeFont ArialMT-Regular, embedded: true
Page 11
    COSName{F23} - PDTrueTypeFont ArialMT-Bold, embedded: true
    COSName{F33} - PDTrueTypeFont ArialMT-BoldItalic, embedded: true
    COSName{F27} - PDTrueTypeFont ArialMT-Regular, embedded: true
Page 12
    COSName{F23} - PDTrueTypeFont ArialMT-Bold, embedded: true
    COSName{F25} - PDTrueTypeFont ArialMT-Italic, embedded: true
    COSName{F27} - PDTrueTypeFont ArialMT-Regular, embedded: true
Page 13
    COSName{F23} - PDTrueTypeFont ArialMT-Bold, embedded: true
    COSName{F25} - PDTrueTypeFont ArialMT-Italic, embedded: true
    COSName{F27} - PDTrueTypeFont ArialMT-Regular, embedded: true
Page 14
    COSName{F23} - PDTrueTypeFont ArialMT-Bold, embedded: true
    COSName{F25} - PDTrueTypeFont ArialMT-Italic, embedded: true
    COSName{F27} - PDTrueTypeFont ArialMT-Regular, embedded: true

Any help appreciated ...

Answer 1

The code in this answer is an attempt to optimize documents like the OP's example document, i.e. documents containing copies of exactly identical objects, in the case at hand completely identical, fully embedded fonts. It does not merge merely nearly identical objects, e.g. multiple subsets of the same font into one single union subset.

In the course of comments to the questions it became clear that the duplicate fonts in the OP's PDF indeed were identical full copies of a source font file. To merge such duplicate objects, one has to collect the complex objects (arrays, dictionaries, streams) of a document, compare them with each other, and then merge duplicates.

As actual pairwise comparison of all complex objects of a document can take too much time in case of large documents, the following code calculates a hash of these objects and only compares objects with identical hash.

To merge duplicates, the code selects one of the duplicates and replaces all references to any of the other duplicates with a reference to the chosen one, removing the other duplicates from the document object pool. To do this more effectively, the code initially not only collects all complex objects but also all references to each of them.

The optimization code

This is the method to call to optimize a PDDocument:

public void optimize(PDDocument pdDocument) throws IOException {
    Map<COSBase, Collection<Reference>> complexObjects = findComplexObjects(pdDocument);
    for (int pass = 0; ; pass++) {
        int merges = mergeDuplicates(complexObjects);
        if (merges <= 0) {
            System.out.printf("Pass %d - No merged objects\n\n", pass);
            break;
        }
        System.out.printf("Pass %d - Merged objects: %d\n\n", pass, merges);
    }
}

(OptimizeAfterMerge method under test)

The optimization takes multiple passes as the equality of some objects can only be recognized after duplicates they reference have been merged.

The following helper methods and classes collect the complex objects of a PDF and the references to each of them:

Map<COSBase, Collection<Reference>> findComplexObjects(PDDocument pdDocument) {
    COSDictionary catalogDictionary = pdDocument.getDocumentCatalog().getCOSObject();
    Map<COSBase, Collection<Reference>> incomingReferences = new HashMap<>();
    incomingReferences.put(catalogDictionary, new ArrayList<>());

    Set<COSBase> lastPass = Collections.<COSBase>singleton(catalogDictionary);
    Set<COSBase> thisPass = new HashSet<>();
    while(!lastPass.isEmpty()) {
        for (COSBase object : lastPass) {
            if (object instanceof COSArray) {
                COSArray array = (COSArray) object;
                for (int i = 0; i < array.size(); i++) {
                    addTarget(new ArrayReference(array, i), incomingReferences, thisPass);
                }
            } else if (object instanceof COSDictionary) {
                COSDictionary dictionary = (COSDictionary) object;
                for (COSName key : dictionary.keySet()) {
                    addTarget(new DictionaryReference(dictionary, key), incomingReferences, thisPass);
                }
            }
        }
        lastPass = thisPass;
        thisPass = new HashSet<>();
    }
    return incomingReferences;
}

void addTarget(Reference reference, Map<COSBase, Collection<Reference>> incomingReferences, Set<COSBase> thisPass) {
    COSBase object = reference.getTo();
    if (object instanceof COSArray || object instanceof COSDictionary) {
        Collection<Reference> incoming = incomingReferences.get(object);
        if (incoming == null) {
            incoming = new ArrayList<>();
            incomingReferences.put(object, incoming);
            thisPass.add(object);
        }
        incoming.add(reference);
    }
}

(OptimizeAfterMerge helper methods findComplexObjects and addTarget)

interface Reference {
    public COSBase getFrom();

    public COSBase getTo();
    public void setTo(COSBase to);
}

static class ArrayReference implements Reference {
    public ArrayReference(COSArray array, int index) {
        this.from = array;
        this.index = index;
    }

    @Override
    public COSBase getFrom() {
        return from;
    }

    @Override
    public COSBase getTo() {
        return resolve(from.get(index));
    }

    @Override
    public void setTo(COSBase to) {
        from.set(index, to);
    }

    final COSArray from;
    final int index;
}

static class DictionaryReference implements Reference {
    public DictionaryReference(COSDictionary dictionary, COSName key) {
        this.from = dictionary;
        this.key = key;
    }

    @Override
    public COSBase getFrom() {
        return from;
    }

    @Override
    public COSBase getTo() {
        return resolve(from.getDictionaryObject(key));
    }

    @Override
    public void setTo(COSBase to) {
        from.setItem(key, to);
    }

    final COSDictionary from;
    final COSName key;
}

(OptimizeAfterMerge helper interface Reference with implementations ArrayReference and DictionaryReference)

And the following helper methods and classes finally identify and merge duplicates:

int mergeDuplicates(Map<COSBase, Collection<Reference>> complexObjects) throws IOException {
    List<HashOfCOSBase> hashes = new ArrayList<>(complexObjects.size());
    for (COSBase object : complexObjects.keySet()) {
        hashes.add(new HashOfCOSBase(object));
    }
    Collections.sort(hashes);

    int removedDuplicates = 0;
    if (!hashes.isEmpty()) {
        int runStart = 0;
        int runHash = hashes.get(0).hash;
        for (int i = 1; i < hashes.size(); i++) {
            int hash = hashes.get(i).hash;
            if (hash != runHash) {
                int runSize = i - runStart;
                if (runSize != 1) {
                    System.out.printf("Equal hash %d for %d elements.\n", runHash, runSize);
                    removedDuplicates += mergeRun(complexObjects, hashes.subList(runStart, i));
                }
                runHash = hash;
                runStart = i;
            }
        }
        int runSize = hashes.size() - runStart;
        if (runSize != 1) {
            System.out.printf("Equal hash %d for %d elements.\n", runHash, runSize);
            removedDuplicates += mergeRun(complexObjects, hashes.subList(runStart, hashes.size()));
        }
    }
    return removedDuplicates;
}

int mergeRun(Map<COSBase, Collection<Reference>> complexObjects, List<HashOfCOSBase> run) {
    int removedDuplicates = 0;

    List<List<COSBase>> duplicateSets = new ArrayList<>();
    for (HashOfCOSBase entry : run) {
        COSBase element = entry.object;
        for (List<COSBase> duplicateSet : duplicateSets) {
            if (equals(element, duplicateSet.get(0))) {
                duplicateSet.add(element);
                element = null;
                break;
            }
        }
        if (element != null) {
            List<COSBase> duplicateSet = new ArrayList<>();
            duplicateSet.add(element);
            duplicateSets.add(duplicateSet);
        }
    }

    System.out.printf("Identified %d set(s) of identical objects in run.\n", duplicateSets.size());

    for (List<COSBase> duplicateSet : duplicateSets) {
        if (duplicateSet.size() > 1) {
            COSBase surviver = duplicateSet.remove(0);
            Collection<Reference> surviverReferences = complexObjects.get(surviver);
            for (COSBase object : duplicateSet) {
                Collection<Reference> references = complexObjects.get(object);
                for (Reference reference : references) {
                    reference.setTo(surviver);
                    surviverReferences.add(reference);
                }
                complexObjects.remove(object);
                removedDuplicates++;
            }
            surviver.setDirect(false);
        }
    }

    return removedDuplicates;
}

boolean equals(COSBase a, COSBase b) {
    if (a instanceof COSArray) {
        if (b instanceof COSArray) {
            COSArray aArray = (COSArray) a;
            COSArray bArray = (COSArray) b;
            if (aArray.size() == bArray.size()) {
                for (int i=0; i < aArray.size(); i++) {
                    if (!resolve(aArray.get(i)).equals(resolve(bArray.get(i))))
                        return false;
                }
                return true;
            }
        }
    } else if (a instanceof COSDictionary) {
        if (b instanceof COSDictionary) {
            COSDictionary aDict = (COSDictionary) a;
            COSDictionary bDict = (COSDictionary) b;
            Set<COSName> keys = aDict.keySet();
            if (keys.equals(bDict.keySet())) {
                for (COSName key : keys) {
                    if (!resolve(aDict.getItem(key)).equals(bDict.getItem(key)))
                        return false;
                }
                // In case of COSStreams we strictly speaking should
                // also compare the stream contents here. But apparently
                // their hashes coincide well enough for the original
                // hashing equality, so let's just assume...
                return true;
            }
        }
    }
    return false;
}

static COSBase resolve(COSBase object) {
    while (object instanceof COSObject)
        object = ((COSObject)object).getObject();
    return object;
}

(OptimizeAfterMerge helper methods mergeDuplicates, mergeRun, equals, and resolve)

static class HashOfCOSBase implements Comparable<HashOfCOSBase> {
    public HashOfCOSBase(COSBase object) throws IOException {
        this.object = object;
        this.hash = calculateHash(object);
    }

    int calculateHash(COSBase object) throws IOException {
        if (object instanceof COSArray) {
            int result = 1;
            for (COSBase member : (COSArray)object)
                result = 31 * result + member.hashCode();
            return result;
        } else if (object instanceof COSDictionary) {
            int result = 3;
            for (Map.Entry<COSName, COSBase> entry : ((COSDictionary)object).entrySet())
                result += entry.hashCode();
            if (object instanceof COSStream) {
                try (   InputStream data = ((COSStream)object).createRawInputStream()   ) {
                    MessageDigest md = MessageDigest.getInstance("MD5");
                    byte[] buffer = new byte[8192];
                    int bytesRead = 0;
                    while((bytesRead = data.read(buffer)) >= 0)
                        md.update(buffer, 0, bytesRead);
                    result = 31 * result + Arrays.hashCode(md.digest());
                } catch (NoSuchAlgorithmException e) {
                    throw new IOException(e);
                }
            }
            return result;
        } else {
            throw new IllegalArgumentException(String.format("Unknown complex COSBase type %s", object.getClass().getName()));
        }
    }

    final COSBase object;
    final int hash;

    @Override
    public int compareTo(HashOfCOSBase o) {
        int result = Integer.compare(hash,  o.hash);
        if (result == 0)
            result = Integer.compare(hashCode(), o.hashCode());
        return result;
    }
}

(OptimizeAfterMerge helper class HashOfCOSBase)

Applying the code to the OP's example document

The OP's example document is about 6.5 MB in size. Applying the above code like this

PDDocument pdDocument = PDDocument.load(SOURCE);

optimize(pdDocument);

pdDocument.save(RESULT);

results in a PDF less than 700 KB in size, and it appears to be complete.

(If something's missing, please tell, I'll try and fix that.)

Words of warning

On one hand this optimizer will not recognize all identical duplicates. In particular in case of circular references duplicate circles of objects won't be recognized because the code only recognizes duplicates if their contents are identical which usually does not happen in duplicate object circles.

On the other hand this optimizer might already be overly eager in some cases because some duplicates might be needed as separate objects for PDF viewers to accept each instance as an individual entity.

Furthermore, this program touches all kinds of objects in the file, even those defining the inner structures of the PDF, but it does not attempt to update any PDFBox classes managing this structure (PDDocument, PDDocumentCatalog, PDAcroForm, ...). To not have any pending changes screw up the whole document, therefore, please only apply this program to freshly loaded, unmodified PDDocument instances and save it soon after without further ado.

Answer 2

When debugging in the file, I recognized that the font files for the same fonts were referenced several times. So replacing the actual font file item in the dictionary with an already viewed font file item, the reference was removed and compression could be done. By that, I was able to shrink a 30 MB File to around 6 MB.

    File file = new File("test.pdf");

    PDDocument doc = PDDocument.load(file);
    Map<String, COSBase> fontFileCache = new HashMap<>();
    for (int pageNumber = 0; pageNumber < doc.getNumberOfPages(); pageNumber++) {
        final PDPage page = doc.getPage(pageNumber);
        COSDictionary pageDictionary = (COSDictionary) page.getResources().getCOSObject().getDictionaryObject(COSName.FONT);
        for (COSName currentFont : pageDictionary.keySet()) {
            COSDictionary fontDictionary = (COSDictionary) pageDictionary.getDictionaryObject(currentFont);
            for (COSName actualFont : fontDictionary.keySet()) {
                COSBase actualFontDictionaryObject = fontDictionary.getDictionaryObject(actualFont);
                if (actualFontDictionaryObject instanceof COSDictionary) {
                    COSDictionary fontFile = (COSDictionary) actualFontDictionaryObject;
                    if (fontFile.getItem(COSName.FONT_NAME) instanceof COSName) {
                        COSName fontName = (COSName) fontFile.getItem(COSName.FONT_NAME);
                        fontFileCache.computeIfAbsent(fontName.getName(), key -> fontFile.getItem(COSName.FONT_FILE2));
                        fontFile.setItem(COSName.FONT_FILE2, fontFileCache.get(fontName.getName()));
                    }
                }
            }
        }
    }

    final ByteArrayOutputStream baos = new ByteArrayOutputStream();
    doc.save(baos);
    final File compressed = new File("test_compressed.pdf");
    baos.writeTo(new FileOutputStream(compressed));

Maybe this is not the most elegant way to do that, but it works and keeps the PDF/A-1b compatibility.