PageValidator: more options / actions
bertsky opened this issue · comments
IMO we should strive to support much more validation and repair features in ocrd_validators.page_validator – esp. functionality known from PRImA Converter and Validator (PCV) and HTR United VX (HTRVX).
From PCV:
-val-rules <Rule1[,Rule2,...]>: Defines what to validate (optional)
Note: If no rules are defined, everthing
is validated.
Available rules (use comma only; no spaces):
General Checks:
VALIDATE_GTSID_DEFINED
VALIDATE_LAYERS
Reading Order:
VALIDATE_READING_ORDER_DEFINED
VALIDATE_READING_ORDER_COMPLETE
VALIDATE_TYPE_OF_REGIONS_IN_READING_ORDER
Region Related Checks:
VALIDATE_REGIONS_WITHIN_DOCUMENT_BOUNDARIES
VALIDATE_REGIONS_WITHIN_BORDER
VALIDATE_REGIONS_DONT_OVERLAP
CALCULATE_REGION_OVERLAP_AREA
VALIDATE_REGION_WITHIN_PARENT_REGION
VALIDATE_NO_INTERSECTING_POLYGON_LINES
VALIDATE_GHOST_REGIONS
VALIDATE_PRINTSPACE
VALIDATE_PENDING_REGIONS
VALIDATE_COMPONENTS_INSIDE_REGIONS (requires image)
VALIDATE_MISSING_ELEMENTS
VALIDATE_NESTED_REGIONS
Text Related Checks:
VALIDATE_TEXT_DEFINED
VALIDATE_UNICODE_TEXT_DEFINED
VALIDATE_DEPRECATED_CHARACTERS
VALIDATE_REPLACEMENT_CHARACTER
VALIDATE_PENDING_CHARACTER
VALIDATE_TEXT_CONTENT
Other:
STRUCTURAL_INTEGRITY
-val-params <INI file>: Load additional validation parameters (optional)
-remove <Filter1[,Filter2,...]>: Remove layout objects (optional).
Available filters (use comma only; no spaces):
REGIONS,NESTED_REGIONS,TEXT_LINES,WORDS,GLYPHS,READING_ORDER,LAYERS
-remove-ghosts <Filter1[,Filter2,...]>: Remove ghost objects (optional)
Ghosts are regions, text lines,
words or glyphs without outline.
Available filters (use comma only; no spaces):
REGIONS,TEXT_LINES,WORDS,GLYPHS,ALL
-convert-text <XML file with rules>: Text content conversion (optional)
-apply-offset <offsetX,offsetY>: Move all layout objects by specified offset
(optional)
Example: -10,20 (no spaces!)
-scale <scaleX,scaleY>: Scale all layout objects by specified factor
Use 'auto' for scaleX and/or scaleY to scale using
the difference between image and XML dimensions.
(optional) (done after apply-offset)
Example: 0.5,0.5 (no spaces!)
-rotate <degrees>: Rotates all polygon points of all layout objects clockwise
around the centre of the page.
-refine-outlines: Refine region outlines. Applies to conversion from non-PAGE
formats (e.g. ALTO) if supported.
General Checks:
VALIDATE_GTSID_DEFINED
trivial
repair: uuid, or rather based on file name?
VALIDATE_LAYERS
Checks if all regions are assigned to
layers (only if there exists at least
one layer).
Not entirely sure we need this, and what it is used for.
Reading Order:
VALIDATE_READING_ORDER_DEFINED
trivial
repair: see page-ensure-readingorder
VALIDATE_READING_ORDER_COMPLETE
Some text regions are missing in the
reading order.
trivial
but not so easy to repair (would require merging existing RO with generated entries)
VALIDATE_TYPE_OF_REGIONS_IN_READING_ORDER
There are one or more regions within
the reading order that shouldn’t be
there (only paragraphs, headings,
drop-capitals, catch-words and TOC-
entries are supposed to be in the
reading order).
trivial
repair: maybe just a filter?
Region Related Checks:
VALIDATE_REGIONS_WITHIN_DOCUMENT_BOUNDARIES
already covered by check_coords
repair: see ocrd-segment-repair
VALIDATE_REGIONS_WITHIN_BORDER
already covered by check_coords
repair: see ocrd-segment-repair
VALIDATE_REGIONS_DONT_OVERLAP
trivial
repair: not trivial, but see ocrd-segment-repair with plausibilize=true and ocrd-cis-ocropy-clip
CALCULATE_REGION_OVERLAP_AREA
trivial
VALIDATE_REGION_WITHIN_PARENT_REGION
already covered by check_coords
repair: see ocrd-segment-repair
VALIDATE_NO_INTERSECTING_POLYGON_LINES
One ore more polygons have
intersecting lines and therefore
contain loops.
i.e. self-intersection
already covered by check_coords
repair: see ocrd-segment-repair
VALIDATE_GHOST_REGIONS
Looks for ghost regions (regions
without outline)
not sure what this exactly means; zero area? negligent area? no coords at all (would already be syntactically invalid)?
VALIDATE_PRINTSPACE
Checks if regions of type page-
number, signature-mark, marginalia
or catch-word are NOT within the
print space
trivial
repair: shrink PrintSpace?
VALIDATE_PENDING_REGIONS
There are text regions without
parent (e.g. word without text line)
How is that even possible syntactically?
VALIDATE_COMPONENTS_INSIDE_REGIONS (requires image)
One ore more regions contain
connected components that are
partly outside the region.
not difficult
repair: see ocrd-cis-ocropy-clip, ocrd-cis-ocropy-resegment (for textlines) and functions postprocess/morphmasks in ocrd-detectron2
VALIDATE_MISSING_ELEMENTS
Some text elements have no child
elements
again, unclear what that entails
Text Related Checks:
VALIDATE_TEXT_DEFINED
Some text regions have no text
ground-truth.
trivial
VALIDATE_UNICODE_TEXT_DEFINED
Some text regions have plain text
defined but not Unicode text.
trivial
VALIDATE_DEPRECATED_CHARACTERS
Deprecated characters are characters that were linked to a private Unicode code point but now have
a dedicated slot in the normal Unicode sections. The filter corrects such changes
doable
VALIDATE_REPLACEMENT_CHARACTER
Checks for occurances of the
replacement character (Unicode
+FFFD) in text element
trivial
VALIDATE_PENDING_CHARACTER
Checks for occurances of pending
characters (Unicode +F51C) in text
elements
trivial
VALIDATE_TEXT_CONTENT
Checks the content of text elements
for inconsistencies (e.g. spaces in
words, trailing line breaks, non
matching text of child and parent
text objects)
already covered by page_textequiv_consistency and page_textequiv_strategy
repair: see page-textequiv-lines-to-regions, page-textequiv-words-to-lines, and function page_update_higher_textequiv_levels in all our *-recognize processors
Other:
STRUCTURAL_INTEGRITY
For XML validation errors, XML
reader warnings (e.g. old PAGE
format) and wrong image
dimensions.
would need to look at the exact implementation, but we should indeed pass any errors from the schema-backed (generateds) parser and present them as (actionable) exceptions
-remove <Filter1[,Filter2,...]>: Remove layout objects (optional).
Available filters (use comma only; no spaces):
REGIONS,NESTED_REGIONS,TEXT_LINES,WORDS,GLYPHS,READING_ORDER,LAYERS
trivial
-remove-ghosts <Filter1[,Filter2,...]>: Remove ghost objects (optional)
Available filters (use comma only; no spaces):
REGIONS,TEXT_LINES,WORDS,GLYPHS,ALL
Ghosts are regions, text lines,
words or glyphs without outline.
see above
-convert-text : Text content conversion (optional)
To apply a filter use -convert-text in the command line call and provide the file name of the XML file
containing the filter rules as additional argument. The XML file must have the following format:<?xml version="1.0" encoding="utf-8"?> <Parameters> <Parameter type="4" name="Replacement Rule" id="1" sortIndex="1" value="0065:=0061" visible="false" isSet="true"> <Description>Replace e by a</Description> </Parameter> <Parameter type="4" name="Replacement Rule" id="2" sortIndex="2" value="0074:=" visible="false" isSet="true"> <Description>Deletes all t</Description> </Parameter> </Parameters>Each parameter element contains a replacement rule. The sortIndex attribute specifies in which
order the rules will be applied. The id attribute must be unique (easiest to use the same value as the
sort index). The description is optional but helps to understand the rules. The actual rule is encoded
in the value attribute. The general format is “HHHH[,HHHH,...]:=[HHHH,HHHH,...]”. HHHH is a
Unicode character represented as 4 digit hexadecimal number. In the example above “0065:=0061”
means ‘replace all characters e with character a’. To replace a character sequence separate the
single characters by comma. The same applies for the right-hand side (the replacement character or
sequence). It is also possible to remove characters by leaving the right-hand side empty (e.g.
“0074:=” to delete all ts).
Sounds like a lot of effort for little gain. People could write their own XSLTs and text processors. But perhaps there are already tons of existing patterns, so supporting this mechanism does have merit?
-apply-offset <offsetX,offsetY>: Move all layout objects by specified offset
(optional)
Example: -10,20 (no spaces!)
trivial
-refine-outlines: Refine region outlines. Applies to conversion from\n");
non-PAGE formats (e.g. ALTO) if supported.\n")
not sure what that entails
From HTRVX:
-s, --segmonto False Apply Segmonto Zoning verification
--zone TEXT None Provide a custom zone to control zone types instead of Segmonto
--line TEXT None Provide a custom line to control Line types instead of Segmonto
tbh, I don't understand that part
-e, --check-empty False Check for empty lines or empty zones
-r, --raise-empty False Warns but not fails if empty lines or empty zones are found
see above
-x, --xsd False Apply XSD Schema verification
see above
-i, --check-image False Check if the image link in the XML points to the right path
already covered