Do analysis of OCR in the PDFs - how to extract
laurensorensen opened this issue · comments
Request from Tom in demo 12/15 - there is a set of PDFs that we received direct from ICJ library, which have OCR, and were ingested for preservation only to SDR. We didn't use the OCR from these documents, but instead made our own and those are what show in the NTA Arclight instance right now.
Here are the complete set of PDF druids from ICJ: https://argo.stanford.edu/catalog?f%5Bcontent_type_ssim%5D%5B%5D=document&f%5Bexploded_tag_ssim%5D%5B%5D=Project+%3A+Nuremberg+trial+archives
I tested a few DRUIDs from the linked set, and none appear to have existing OCR. That is, when opened in Adobe Reader, there was no text available to be selected in the PDF — only images. In order to get text out, I needed to invoke Adobe's own OCR engine.
For comparison, here's one document — "Official Commission Transcript, Walter ALBATH, 10 June 1946: 1946-06-10" — which corresponds to this item in staging, this item in Argo which was OCR'd by us (using ABBYY), and this item in Argo which is the original ICJ-provided PDF.
Here's a selection from our OCR (line breaks added for clarity):
IMT Nuremberg Archives H-937 HO937 -000 International Court of Justice
COM. Take 4- Pg 8 Lesser 10 June 1946 HO937 -0002
OFFICIAL TRANSCRIPT OF TESTIMONY FOR THS DEFENSE OF ORGANIZATIONS, TAKEN BEFORE A COMMISSION APPOINTED BY THE INTERNATIONAL MILITARY TRIBUNAL PURSUANT TO PARAGRAPH 4 OF THE ORDER OF UE TRIBUNAL DATED 13 MARCH 1946.
10 June 1946 OMIISSIONERS: Colonel A.M.S. Neave Captain Tubridy THE CESTAPO (THE COMMISSION CONVENED AT 1600 HOURS.)
WALTER ALBATH, a witness, took the stand and testified as follows:
BY COLONEL NEAVE:
0 What is your name?
A Walter Albath,
Q Will you raise your right hand and repeat this oath after me: I swear by God, the. Almighty and Omniscient, that I will speak the pure truth and will withhold and add nothing.
(The witness repeated the oath.)
COLONEL NEAVE: You may sit down. Dr. Weisgerber, will you tell us what this witness is supposed to prove? You may give us a trans lation of the written statement you have just given me.
DR. WEISGERBER: First I should like to ask this witness concerning the tasks, organizations and activities of the Inspector of the Security Police and SD and second, concerning his activities as to the supervision of the offices of the Gestapo subordinate to him.
And here's the same passage after I invoked Adobe's built-in OCR. This text does not exist in the document in Argo.
1111111111111111111111111111111111111
H0937-0001
IMT Nuremberg Archives
H-937
International Court of Justice
co ..
ifaire' 4-Pg 8 lesser
II I I II IIII I I I II IIII II 11111111111 11 111
10 J,une 1946 H0937-0002
OFFICI i\L TRA SCRIPT OF TSSTI NY FCR THC lJS JS'i'
OF ORGA TIZ1 TI N0, T/tlffi!T BE R /1. COlOOSSION
PPOH .,;,D BY TI� INTIRNI\TIO Ll\L LIT \RY TRIIUNJ\L
PURSU T 'ID Pt RtiGTMPH 4 OF THE ORDER OF 'IHE TRIBUWL DATED 13 LLillCH 1946.
10 June 1946
001 �ITSSIOJ:-!ERS:
Colonel A.� s. eave Captain Tubridy
THE GESTAPO
(THE cc: assrm: corn: ED AT 1600 HOURS.)
YTAL'IER . BATH, a witness, took the stand and testified as follov,s:
BY 0010 EL I �AVE:
0 . hat is your name? A !alter AJ.bath. 'ifill you raise your right hand and repeat this oath after re:
I swear by God, the, Almighty and Omniscient, that I will speak the pure truth and Will withhold and add nothing.
(The .witness repeated the oath.)
COLONEL :r-8/:VE: You may sit down. Dr. �reisgerber, will you
tell us wha. t this witness is supposed to prove? You may give us a trans.lation of the written staten:ent you have just given re.
DR. 1:'IEISGERBER: First I should like to ask this witness concerning
the tasks, organizations and activities of the Inspector of the Security Police and SD and second, concerning his activities as to the supervision
of the offices of the Gestapo subordinate to him.
I think the difference is pretty clear. I checked a few DRUIDs; unless I'm missing something:
- We don't have any OCR provided from ICJ.
- Even if we did, the OCR we generated is much better.