There’s no shortage of articles claiming that one PDF processing library outperforms all others. In fact, it’s not uncommon for multiple vendors to each present evidence that they are the best.

Assuming these benchmarks are conducted honestly, how can this be true?

The answer lies in the details:
what is being tested, how it is measured, and what is meant by “correct”.

A benchmark is only meaningful if:

When a single metric becomes the only thing that matters, systems will optimize for that metric at the expense of real-world performance such as with the Volkswagen Dieselgate scandal, or using macOS to measure OCR when the intended deployment is to a Linux machine.

Designing a representative set of test documents is surprisingly difficult. Small decisions in corpus design can dramatically bias results.

PDFs differ widely depending on their origin:

That’s a huge problem since a tool tuned for invoices will likely perform extremely well on financial data but fail badly on academic papers with multi-column layouts and embedded figures.

It’s therefore difficult to create a have a single corpus that is meaningful for multiple industries.

Many or the corpora use English language documents, and if that is your target audience then that’s fine. But what about if all of some of your market uses another language, whether a Latin-based language such as Spanish, or something entirely different such as Chinese or Thai?

Language affects not just the characters used, but also word boundaries (English uses spaces between words while Chinese does not), and text direction (Arabic reads right to left and Chinese and Japanese often read from top to bottom)

If your benchmark only includes English documents, you’re not measuring global performance.

Knowing how the sample documents were created matters:

Born-digital PDFs

Scanned PDFs

An error in OCR can lead to down-stream document structure misunderstanding. For example, misidentifying a “1” as “I” could mean that a page number is not identified, which could then result in a footer not being detected. An updated OCR engine such as PaddleOCR could result in significant improvement in structure data extraction, even if nothing else in the library changes.

A benchmark that excludes scanned documents can dramatically overestimate real-world accuracy.

Whether that is a problem depends, once again, on what you or your customers need.

A well-tagged PDF includes structural metadata that identify parts of the PDF as (for example):

Some PDF libraries leverage that information to increase accuracy, others don’t. That can make the difference between a table actually being detected as a table rather than as just text.

With the introduction of Accessibility legislation (such as the European Accessibility Act), newly created documents are increasingly likely to use tagging to give PDF/UA. That’s great news for the less able.

Many of the publicly available corpora have a high proportion of PDFs that were collected ten or more years ago. As a consequence, they are disproportionately biased towards untagged documents.

If your test set uses only tagged PDFs, then some SDKs might give far better results than users would experience with a set of older, untagged, documents.

On the other hand, if the test set just uses untagged PDFs, then it may biased the results towards less sophisticated tools, which give worse results that other libraries with modern, tagged, documents.

PDFs can, of course, represent just about anything that can be printed.

They could be a half a page long, or hundreds of pages long

Each of these introduces opportunities for extraction errors. The longer the document the more opportunity there is to go wrong.

Short is simple, but very often it is not representative of real-world documents.

A benchmark that uses just single page documents, arguably has no real world value.

Having realistic original documents is one thing, but length of a sample file is another.

Many corpora contain files that are just single pages taken from a multi-page PDF. That makes life simple (as we saw with short documents), but it has more subtle effects too. If you only have a single page taken from a larger document, then the data extraction library cannot use context from surrounding pages (since they have been removed).

As a result:

That biases results towards tools that are good at working on a single page in isolation, but biases against smarter tools that are context aware (such as the Apryse SDK).

Many of the corpora used for benchmarking such as dp-bench (the corpus that Nutrient used), are therefore of limited value.

A further problem with testing against single page documents is that it gives you no idea whether the tool can work with real, longer documents.

A tool with poor memory management might work well with a single page document but run out of memory, or slow down dramatically,when faced with a 50 page business report?

Having a corpus of files that the software vendor won’t share implies “trust me, we absolutely wouldn’t say things just to get your money.”

But not all files can be shared. They might

Creating a corpus of documents that is realistic, and shareable, is therefore extremely hard.

Finally, the sample set should be large enough to cover a range of scenarios that you care about, but small enough that it can be run quickly.

Ultimately there is no point having 400 PDFs that are essentially the same. That just makes testing slow (and gives a lot of data to review).

But for a general corpus there is a good reason to have a mix of PDFs from different industries, with both scanned and born digital files. Typical corpus size for benchmarking is around 200 files, which is probably a good (and understandable size). However, some corpora (for example DP-Bench) have several documents that are single pages taken from the same original PDF, and likely are not testing anything different.

Corpus creation is therefore extremely complex, with a large number of trade-offs if a “One size fits all” corpus is required.

It is one reason why you should use benchmark data only as a starting point and then test with your own documents.

Knowing which files to test is only part of the problem when comparing different vendors. You also need to be able to compare the outputs to work out which one gives the best results.

Not all libraries produce the same output:

These formats are not equivalent.

For example, let’s consider this page from a PDF:

Markdown output for the text within a footnote would be simple text.

Whereas JSON output could contain information about location and font size:

Or even with more information about being a footnote, or italic etc.

All of these outputs are “true”, but some contain structural information that is lost elsewhere (particularly when exporting to markdown).

Ultimately, the ‘Correct” answer depends on which one solves the user’s particular problem. Markdown might be good enough, or the discarded information might be a showstopper.

This makes direct comparisons between tools inherently difficult, and it gets harder.

To measure accuracy, you need a ground truth. But PDFs are fundamentally ambiguous.

Sometimes there is a single, obvious way to evaluate what the content of a PDF and the correct reading order is. On other occasions that is not the case. Consider one of the files (which I chose at random) from the DP-Bench corpus.

Similarly, reading order of a PDF can be ambiguous.

As a human I would probably read most of the first column, then jump to a footnote, then look at the image, then go back to the first column, then onto the second column, jump to a footnote, spot that there is another footnote that is from the same source, then go back to the text.

But if I was to read the document again then I would likely do something different.

And should the footnotes for the first column, if we include them at all, go after the first column’s text, or after all of the text?

Similarly, if there is a footer, then I would probably skip over the footer, but only if I recognize it as a footer.

Creating “Ground truth” is therefore hard.

Even if the “truth” is non-ambiguous generating it is tedious.

It’s almost certainly one of the reasons why many benchmark corpora use single page documents.

In the next article we will look at how Apryse created a publicly available corpus for benchmarking Smart Data Extraction.

If you have questions or want to discuss benchmarking strategies, reach out via our http://www.apryse.com/discord or email blog-feedback@apryse.com.